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FOREWORD 


The  Center  for  Army  Leadership  (CAL)  and  the  U.S.  Army  Research  Institute  for  the  Behavioral 
and  Social  Sciences  (ARI)  joined  together  to  identify  what  it  takes  to  be  a  competent  Army  leader  and 
battle  commander.  The  conceptual  skills  of  officers  have  been  of  particular  interest.  There  has  been 
remarklablv  little  attention  given  to  higher  conceptual  skills  in  Army  instruction  and  personal  development 
programs.  Researchers  from  our  two  activities  examined  the  relationship  of  conceptual  capacity  and  battle 
command  performance.  The  specific  behavior  that  was  investigated  was  the  ability  to  discern  critical 
information.  Six  sublevel  behaviors,  which  were  believed  to  contribute  to  critical  information  discernment, 
were  identified  by  the  research  team.  Fifty  Command  and  General  Staff  Officer  College  students  were 
rated  by  experienced  observers  during  Prairie  Warrior  ‘96.  These  ratings  were  correlated  with  measures  of 
conceptual  capacity.  It  was  hypothesized  that  individuals  with  higher  conceptual  capacity  would  display 
higher  levels  of  critical  information  discernment. 

The  research  underscored  the  value  of  ties  between  the  ARI  and  the  CAL.  A  combination  of 
psychological  tests  of  cognition  and  behavioral  rating  of  performance  provided  useful  insights  into 
measuring  conceputal  capacity  and  critical  information  discernment.  The  findings,  including  the  lack  of 
some  expected  relationships,  draw  out  the  importance  of  verifying  beliefs  about  factors  and  traits 
contributing  to  leadership  performance.  Subsequent  research  is  continuing  to  examine  additional  individual 
differences  in  battle  command  performance.  With  a  better  understanding  of  the  components  of  successful 
battle  command,  the  Army  will  be  better  able  to  push  forward  with  development  and  training  plans  for 
Force  XXI  and  Army  After  Next  leaders. 


JOHN  A.  SPEARS 
Colonel,  Infantry 
Director,  Center  for  Army 
Leadership 


ZITA  M.  SIMUTIS 
Technical  Director 
Army  Research  Institute 


EDGAR  M.  JOHNSON 
Director 

Army  Research  Institute 
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DISCERNING  CRITICAL  INFORMATION:  A  PRAIRIE  WARRIOR  ‘96  CASE  STUDY 


EXECUTIVE  SUMMARY 


Research  Requirement: 

Can  the  U.S.  Army  identify  a  means  of  selecting  or  developing  leaders  who  have  the  ability  to 
discern  critical  information?  Findings  from  previous  Advanced  Warfighter  Experiments  (AWE), 
Warfighter  Exercises  (WFX),  and  Combat  Training  Center  (CTC)  rotations  indicated  that  leaders  were  not 
improving  in  their  abilities  to  determine  what  information  was  relevant  and  how  to  properly  package  and 
disseminate  that  information  to  the  appropriate  level.  A  leader's  discernment  ability  becomes  increasingly 
important  as  the  Army  becomes  smaller  and  its  missions  extend  to  nontraditional  roles  of  operations  other 
than  war.  While  information  technologies  increase  an  organization’s  capacity  for  handling  predictable 
information,  the  requirement  for  leaders  to  digest  and  act  on  more  complex  information  and  to  handle 
unpredictable  situations  increases. 

Emerging  leadership  doctrine  identifies  three  categories  of  competencies:  technical  and  tactical, 
interpersonal  and  communicative,  and  conceptual  (Headquarters,  DA,  1987).  The  Army  has  invested  a 
significant  amount  of  resources  designing  formal  programs  that  develop  the  first  two  categories  and 
virtually  no  time  considering  conceptual  competencies,  let  alone  focusing  assessment  or  development  on 
them  (Kluever,  Lynch,  Matthies,  Owens  &  Spears,  1992).  The  intent  of  this  study  is  to  begin  to  make  up 
for  this  imbalance. 

During  the  preparation  of  the  experiment  plan  for  Prairie  Warrior  96  (PW96),  it  became  apparent 
that  although  various  information  technologies  would  be  tested  in  the  exercise,  there  were  no  plans  to  study 
the  conceptual  capacities  related  to  the  technologies.  The  Battlefield  Visualization  Experiment  in  the  PW96 
AWE  corrected  this  oversight.  It  provided  an  opportunity  for  researchers  from  the  Center  for  Army 
Leadership  (CAL),  U.S. Army  Research  Institute  for  the  Behavioral  and  Social  Sciences  (ARI)  and 
Industrial  College  of  the  Armed  Forces  (ICAF)  to  test  the  relationship  between  cognitive  complexity,  stated 
in  this  report  as  conceptual  capacity,  and  commander  and  staff  performance  in  discerning  critical 
information.  This  case  study  was  developed  to  explore  whether  indicators  of  conceptual  capacity  are 
predictive  of  critical  information  discernment  in  the  context  of  battlefield  visualization. 

Procedure: 

The  primary  hypothesis  of  the  study  was  that:  Individuals  with  higher  conceptual  capacity  would 
be  better  able  to  identify,  synthesize,  package,  and  disseminate  critical  information.  A  secondary  purpose 
of  the  study  was  to  explore  the  development  of  methods  for  the  study  of  leadership  traits  and  the  human 
dimensions  of  battlefield  visualization.  Supporting  objectives  were  to  try  out  various  measurement 
instruments  and  to  gather  psychometric  data  on  them . 

This  case  study  was  structured  to  assess  current  level  of  conceptual  capacity  and  critical 
information  discernment  (CID)  behaviors.  Conceptual  capacity  level  was  measured  before  the  exercise 
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using  a  modified  version  of  the  Career  Path  Appreciation  (CPA)  method  and  a  Biodata  instrument  from 
ARI.  Results  were  interpreted  at  ICAF  and  returned.  Critical  information  discernment  behaviors  were 
assessed  during  PW96  by  observation.  Observers  were  provided  a  score  sheet  and  used  it  to  make  daily 
observations.  Additionally,  critical  information  discernment  behavior  was  assessed  from  peer  and  self 
ratings  collected  as  supplemental  items  on  an  ARI  360  degree  assessment  instrument  called  Azimuth.  The 
data  sets  for  conceptual  capacity  level  and  critical  infonnation  discernment  behaviors  were  then  compared 
to  determine  if  a  relationship  existed.  Interviews  with  selected  participants  were  conducted  to  see  if  there 
were  any  distinguishing  work  or  educational  experiences  among  high  and  low  performers. 

Two  groups  of  students  from  the  Command  and  General  Staff  Officer  Course  (CGSOC)  were 
earmarked  for  participation  in  this  study.  One  group  consisted  of  4 1  officers  from  the  Mobile  Strike  Force 
(MSF).  Officers  were  earmarked  for  participation  based  on  the  key  positions  they  held.  The  second  group 
consisted  of  27  students  portraying  commanders  and  staff  officers  of  II  Corps  (Student  Corps).  The  68 
made  up  the  initial  pool  of  individuals  who  agreed  to  fill  out  various  survey  instruments  and  to  be  observed 
during  PW96. 

Findings: 

There  was  no  direct  relationship  between  the  modified  CPA  (MCPA)  measure  of  Conceptual 
Capacity  Level  and  ranking  of  CID  performance  (r=  -.07)  (see  Figure  1).  There  were  moderate  negative 
correlations  between  CID  performance  and  Cognitive  Complexity  (r=  -.25,  p=  .09)  and  Openness  scales 
(r=  -.28,  p=  .05)  from  a  Biodata  test.  The  relationship  between  Critical  Information  Discernment  and 
Openness  were  stronger  (negatively)  when  differences  in  jobs,  difficulty,  experience,  and  time  pressure 
were  accounted  for  (r=  -.43). 

There  were  high  correlations  whenever  measures  were  based  on  self-report  instruments  (MCPA, 
Biodata,  Azimuth).  An  individual's  bias  in  response  was  consistent  across  self-report  instruments. 
Experienced  observer's  ratings  of  performance  did  not  correspond  to  participant’s  ratings  of  performance. 
Additionally,  neither  of  these  corresponded  to  peers’  ratings  of  performance.  The  findings  suggest  one  or 
more  of  the  following:  (1)  it  is  difficult  to  rely  on  individuals  to  reliably  and  validly  rate  themselves;  (2)  it 
is  difficult  to  test  for  conceptual  capacity;  and  (3)  it  is  difficult  to  observe  CID  skill.  A  trend  was  noticed 
from  the  Critical  Information  discernment  score  sheets  that  indicated  high  infonnation  discemers  were  more 
willing  to  risk  making  mistakes  and  more  confident  in  their  assessments/predictions  than  those  that  made 
fewer  assessments/predictions. 

Additional  examination  of  the  psychometric  parameters  of  the  Conceptual  Capacity  Level  variable 
seems  to  be  in  order.  A  previous  study  (Lewis,  1993)  showed  modest  levels  of  construct  validity  (.42  to 
.46)  for  the  CPA  approach  for  measuring  Conceptual  Capacity  Level.  Although  these  validity  levels  are 
somewhat  supportive,  they  are  not  overwhelming  evidence  that  Conceptual  Capacity  Level  is  a  dependable 
predictor  at  present.  The  CPA  was  modified  for  this  study  to  focus  on  the  Phrases  portion,  which  allows 
the  individual  to  make  choices  about  phrases  that  describe  his  or  her  style  of  working.  Nonstandard 
interpretation  of  the  phrases  by  students  and  responses  affected  by  social  desirability  certainly  seem 
possible.  Other  measurement  methods  should  be  identified  (or  developed)  and  tried.  There  should  be  more 
emphasis  on  requiring  the  individual  to  demonstrate  abilities  as  opposed  to  attempt  to  describe  processes 
about  which  he  or  she  may  not  be  familiar. 

Measuring  information  discernment  performance  by  observation  was  difficult.  As  far  as  we  know, 
this  was  the  first  time  that  CID  behaviors  had  been  measured  in  this  way.  As  a  result,  normative  or 
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doctrinal  standards  did  not  exist  for  comparison.  Another  limitation  was  that  observers  could  have  differed 
in  how  they  decided  to  give  a  rating  to  a  student  they  were  observing.  These  differences  were  accounted  for 
through  observer  training  and  by  having  members  of  the  research  team  observe  both  the  students  and  the 
observers  in  the  performance  of  their  duties.  We  found  the  narrative  explanation  on  the  score  sheet 
consistently  provided  the  most  insight  into  a  student’s  information  discernment  performance.  The  narrative 
was  the  primary  source  of  information  used  by  a  panel  to  rank  order  the  students’  CID  performance. 

The  score  sheet  enabled  researchers  to  discriminate  among  students’  information  discernment 
behaviors  and  did  not  impose  an  excessive  workload  on  the  observers.  It  has  potential  for  application 
during  scheduled  WFXs  and  CTC  rotations. 

From  this  study  we  learned  additionally  the  importance  of  measuring  knowledge,  experience,  and 
motivation.  A  very'  plausible  hypothesis— but  untestable  with  the  current  data— is  that  experience  or 
knowledge  had  a  considerable  influence  on  performance.  Differences  in  experience  could  have 
overshadowed  other  differences  or  effects  of  the  hypothesized  relationship  with  cognitive  complexity.  It 
seems,  based  on  limited  data,  that  low  student  motivation  did  not  mask  a  relationship  between  performance 
and  Conceptual  Capacity’  Level.  We  observed  instances  of  low  participation,  but  the  effect  was  not 
universal.  Motivated  and  challenged  students  tended  to  perform  better,  but  were  not  necessarily  higher 
scorers  on  cognitive  complexity.  Motivation  and  experience  should  be  more  completely  measured  in  future 
studies  in  this  area. 

Utilization  of  Findings: 

The  findings  in  this  study  highlight  the  complexity  involved  in  understanding  the  cognitive  aspects 
of  human  behavior  and  the  intricate  relationships  among  the  factors.  The  findings  emphasize  the  need  for 
increased  effort  to  understand  and  define  the  conceptual  skills  that  are  desired  in  Army  leaders.  Measuring 
conceptual  capacity  in  a  way  that  requires  individuals  to  demonstrate  their  ability'  provides  a  possible  focus 
for  future  research  and  training  development  efforts.  Follow-up  studies  are  essential  to  address  these 
additional  aspects  of  the  human  dimension  of  Battlefield  Visualization  and  to  assist  in  the  design  of 
strategies  that  address  the  development  of  conceptual  competencies.  The  best  possible  outcome  from  this 
study  would  be  that  it  energizes  multiple  efforts  for  improved  leader  development  programs. 
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DISCERNING  CRITICAL  INFORMATION:  A  PRAIRIE  WARRIOR  '96  CASE  STUDY 

Introduction 


The  Problem 

Can  the  US  Army  identify  a  means  of  selecting  or  developing  leaders  who  have  the  ability  to 
discern  critical  information?  Findings  from  previous  Advanced  Warfighter  Experiments  (AWE), 
Warfighter  Exercises  (WFX),  and  Combat  Training  Center  (CTC)  rotations  indicated  that  leaders  were  not 
improving  in  their  abilities  to  determine  what  information  was  relevant  and  how  to  properly  package  and 
disseminate  that  information  to  the  appropriate  level.  A  leader's  discernment  ability  becomes  increasingly 
important  as  the  Army  becomes  smaller  and  its  missions  extend  to  nontraditional  roles  of  operations  other 
than  war.  While  information  technologies  increase  an  organization’s  capacity  for  handling  predictable 
information,  the  requirement  for  leaders  to  digest  and  act  on  more  complex  information  and  to  handle 
unpredictable  situations  increases. 


Background 

Emerging  leadership  doctrine  identifies  three  categories  of  competencies:  technical  and  tactical, 
interpersonal  and  communicative,  and  conceptual  (Headquarters,  DA,  1987).  The  Army  has  invested  a 
significant  amount  of  resources  designing  formal  programs  which  develop  the  first  two  categories  and 
virtually  no  time  considering  conceptual  competencies,  let  alone  focusing  assessment  or  development  on 
them  (Kluever,  Lynch,  Matthies,  Owens  &  Spears,  1992).  The  intent  of  this  study  is  to  begin  to  make  up 
for  this  imbalance. 

During  the  preparation  of  the  experiment  plan  for  Prairie  Warrior  96  (PW96),  it  became  apparent 
that  although  various  information  technologies  would  be  tested  in  the  Mobile  Strike  Force  (MSF) 1  no 
study  of  the  conceptual  capacities  related  to  the  technologies  would  be  addressed. 

One  of  the  MSF  initiatives  this  year  was  the  use  of  a  Combat  Information  Center  (CIC)  to  produce 
"a  single,  synthesized  picture  of  friendly  and  enemy  forces,  and  the  battlespace  environment"  called  the 
relevant  common  picture  (RCP)  (p.  3-20,  EER  Systems,  1996).  The  CIC  was  to  serve  as  a  centralized 
collection  point  for  battlefield  intelligence  from  across  functional  areas  and  to  integrate  the  information  into 
a  meaningful  situation  picture  for  the  Commander  and  others.  This  MSF  initiative  was  seen  as  an  excellent 
opportunity  to  study  how  individuals  who  receive  critical  information  understand  and  integrate  that 
information.  Not  everyone  in  a  command  post  recognizes  the  criticality  of  information  or  interprets  it  in  the 
same  way.  The  competency  that  was  thought  to  be  key  to  the  operation  of  the  CIC  was  critical  information 
discernment,  a  key  component  in  battlefield  visualization. 

People  differ  in  many  ways  in  how  they  discern  information.  Some  individuals  understand 
information  in  a  very  concrete  way,  considering  only  the  meaning  at  the  surface.  Others  go  beyond  the 
apparent  meaning  and  make  second  and  third  order  interpretations  and  associations  with  what  else  they 
know  (Fredericksen,  1986;  Holyoak  &  Nisbett,  1988;  Soloway,  Adelson  &  Ehrlich,  1988).  People  also 
differ  in  preference  for  detail  or  for  general  information  (Michel  &  Riedel,  1988).  Some  people  operate  at 


1  The  MSF  is  a  notional  division  used  during  Prairie  Warrior  exercises  to  explore  and  develop  doctrine  and 
capabilities  for  future  forces. 
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a  theoretical  level  while  others  operate  at  an  experiential  and  applied  level  (Kolb,  1984).  Besides  differing 
among  these  characteristics,  each  individual  is  likely  to  adapt  his  or  her  problem  solving  approach  to  the 
situation  (Klein,  Orasanu,  Calderwood,  &  Zsambok,  1993;  Payne,  Bettman  &  Johnson,  1993). 

One  relevant  approach  that  tries  to  account  for  individual  differences  is  cognitive  complexity. 
Cognitive  complexity 2  involves  consistent  tendencies  of  individuals  to  process  information  in  certain  ways. 
A  simplistic  view  of  cognitive  complexity  is  the  extent  of  the  breadth  and  depth  in  the  way  a  person  thinks. 
The  concept  of  cognitive  complexity  is  marked  by  two  cognitive  processes:  differentiation  and  integration. 
Differentiation  is  the  ability  to  discern  multiple  dimensions  or  differences,  and  integration  is  the  ability  to 
comprehend  multiple  concepts  and  put  them  together  in  a  meaningful  way.  Sometimes  cognitive  complexity 
is  discussed  as  work  capacity.  The  longer  the  time  span  that  one  sees  and  plans  for,  the  higher  one's  work 
capacity  (Hooijberg  &  Quinn,  1992).  The  theory  implies  that  an  individual  with  higher  cognitive 
complexity  can  understand  complex  situations  and  is  better  able  to  make  good  decisions  in  uncertain 
situations. 

Jacques  (1976)  used  the  notion  of  cognitive  complexity  in  his  stratified  systems  theory  (SST). 

SST  proposes  that  the  higher  the  level  of  an  organization,  the  more  complex  the  tasks  and  functions.  As 
functions  increase  in  complexity  so  must  the  responsible  leaders  capability  for  complexity  increase.  SST 
specifies  that  the  number  of  levels  will  not  exceed  seven  for  any  organization.  Jacques  (1989)  has 
determined  various  parameters  for  the  different  "strata"  of  organizations  and  Jacobs  and  Jacques  (1987) 
have  specified  the  associated  personnel  requirements.  Time  span  is  one  way  to  measure  complexity  of 
work.  Time  span  is  the  maximum  allowed  completion  time  of  the  longest  tasks  or  programs  for  which  an 
individual  is  held  accountable. 

One  approach  to  measure  an  individual's  potential  for  complex  work  is  Career  Path  Appreciation 
(CPA)  (Stamp,  1986).  CPA  has  had  some  success  predicting  managerial  performance  (Stamp,  1988)  in 
prior  research.  Although  this  was  not  directly  related  to  leadership,  it  did  provide  insights  into  the 
difficulty  of  predicting  leadership  potential.  A  predictive  capability  for  Army  leaders  would  have  merit 
because  leadership  potential  could  be  identified  early,  and  officer  assignments  could  be  optimized  to  further 
develop  or  to  take  advantage  of  the  identified  capabilities.  Although  desirable,  the  ability  to  predict 
leadership  reliably  is  probably  many  years  into  the  future.  An  ability  to  predict  high  performing  leaders 
also  would  imply  that  an  individual's  level  of  complexity  at  any  point  in  their  careers  could  be  determined. 
If  cognitive  capacity  can  be  reliably  measured,  then  we  can  see  if  capacity  is  a  stable  characteristic  or 
whether  it  might  be  increased  through  training  (e.g.  Streufert  &  Swezey,  1986). 

PW96  provided  an  opportunity  to  test  the  relationship  between  cognitive  complexity,  stated  in  this 
report  as  conceptual  capacity,  and  commander  and  staff  performance  in  discerning  critical  information. 

This  case  study  was  developed  to  explore  whether  indicators  of  conceptual  capacity  are  predictive  of 
critical  information  discernment  in  the  context  of  battlefield  visualization. 


2  There  is  not  room  in  this  report  to  give  complete  background  on  cognitive  complexity  and  its  relationship  to  other 
variables.  For  excellent  reviews  refer  to  McDaniel  &  Lawrence,  1990;  Streufert  &  Swezey,  1986;  and  Streufert  & 
Streufert,  1978.  For  proposed  relationships  to  leadership  refer  to  Jacobs  &  Jacques,  1987;  Jacques  &  Clement, 
1991;  Phillips  &  Hunt,  1992. 
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Method 


Hypothesis 

The  primary  hypothesis  of  the  study  was  that:  Individuals  with  higher  conceptual  capacity  would 
be  better  able  to  identify,  synthesize,  package  and  disseminate  critical  information.  A  secondary  purpose 
of  the  study  was  to  explore  the  development  of  methods  for  the  study  of  leadership  traits  and  the  human 
dimensions  of  battlefield  visualization.  Supporting  objectives  were  to  try  out  various  measurement 
instruments  and  to  gather  psychometric  data  on  them. 


Case  Study  Structure 

This  case  study  was  structured  to  assess  current  level  of  conceptual  capacity  and  critical 
information  discernment  behaviors.  Conceptual  capacity  level  was  measured  before  the  exercise  using  a 
modified  version  of  the  CPA  method  and  a  Biodata  instrument  from  ARI.  CPA  results  were  interpreted  at 
the  Industrial  College  of  the  Armed  Forces  (ICAF)  and  returned.  Critical  information  discernment 
behaviors  were  assessed  during  PW96  by  observation.  Observers  were  provided  a  score  sheet  (see 

Appendix  A)  and  used  it  to  make  daily  observations.  Additionally,  critical  information  discernment 
behavior  was  assessed  from  peer  and  self  ratings  collected  as  supplemental  items  on  an  ARI  360  degree 
assessment  instrument  called  Azimuth.  The  data  sets  for  conceptual  capacity  level  and  critical  information 
discernment  behaviors  were  then  compared  to  determine  if  a  relationship  existed.  Interviews  with  selected 
participants  were  conducted  to  see  if  there  were  any  distinguishing  work  or  educational  experiences  among 
high  and  low  performers. 


Participants 

Two  groups  of  students  from  the  Command  and  General  Staff  Officer  Course  (CGSOC)  were 
earmarked  for  participation  in  this  study.  One  group  consisted  of  41  officers  from  the  MSF.  Officers  were 
earmarked  for  participation  based  on  the  key  positions  they  held.  The  second  group  consisted  of  27  students 
portraying  commanders  and  staff  officers  of  II  Corps  (Student  Corps).  Different  numbers  of  participants 
were  available  for  the  various  surveys  and  observation  during  PW96.  (See  Appendix  B  for  the  number  of 
MSF  and  II  Corps  participants  for  each  set  of  measurement  instruments). 


Assessment  Instruments 

Modified  CPA  The  CPA  method  consists  of  three  parts:  symbols3 ,  work  history4 ,  and  phrases 
(Lewis,  1993;  Stamp,  1986).  In  the  phrases  portion,  an  interviewee  receives  nine  sets  of  six  cards  with 
phrases  about  a  style  of  working  or  thinking.  The  interviewee  selects  the  phrase  which  is  most  like  how  he 
or  she  works  and  the  phrase  which  is  the  least  preferred  way  to  approach  work.  The  interviewee  also 


3  The  symbols  task  involves  the  Bruner  concept  formation  task  in  which  the  interviewee  is  to  discover  a  rule  by 
which  a  set  of  symbols  depicted  on  cards  is  sorted  according  to  four  example  "targets’1  (Bruner,  1966). 

4  The  work  history  involves  surveying  the  interviewee's  major  work  assignments  and  their  challenge,  time  span, 
and  time  horizon. 
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elaborates  on  the  choices.  The  phrases  are  ordered  from  less  complex  to  more  complex.  The  average  score 
of  most  preferred  selections  was  used  as  a  starting  indicator  of  current  conceptual  capacity  level,  to  which, 
clinical  judgment  of  semantic  content  was  applied,  with  an  eye  on  total  inversions  (number  of  instances 
where  the  least  preferred  phrase  was  higher  than  the  most  preferred  phrase).  The  CPA  results  in  a  score  of 
current  work  capacity  and,  with  the  interviewee's  age,  a  score  of  predicted  future  work  capacity  or  "mode" 
(Lewis,  1993). 

For  this  study,  the  phrases  portion  of  the  CPA  was  administered  to  41  students  from  the  MSF  and 
27  students  from  II  Corps.  Materials  from  two  of  the  MSF  students  and  one  of  the  II  Corps  students  were 
not  scorable.  There  were  two  scorers  from  ICAF  who  assigned  current  work  capacity  or  Conceptual 
Capacity  Level  (CL),  which  was  the  primary  predictor  in  this  study.  Scoring  requires  clinical  judgment 
applied  by  trained  interviewees.  The  Conceptual  Capacity  Level  measure  was  the  midpoint  of  the  two 
scores.  The  scorers  used  the  following  in  their  judgments:  the  sum  of  point  values  of  all  "preferred" 
phrases,  the  number  of  times  a  less  complex  phrase  was  preferred  over  a  more  complex  one,  the  average 
number  of  lines  written  to  explain  die  nine  sets  of  "most"  and  "least"  choices,  and  the  content  of  the 
interviewees  explanation. 

In  a  previous  study  (Lewis,  1993),  the  CPA  had  an  inter-rater  reliability  of  .81  .  The  CPA  also 
had  a  correlation  of  .46  with  two  Army  War  College  (AWC)  instructors’  ratings  of  the  students’  potential 
to  become  general  officers.  Initial  characteristics  of  the  modified  CPA  were  available  from  a  separate 
sample  of  ICAF  students  (McGee,  Jacobs,  Kilcullen  &  Barber,  1996).  The  correlation  between  the 
modified  CPA  and  the  CPA  was  .68.  On  the  average,  the  modified  CPA  was  about  1 .5  points  lower  than 
the  CPA.  The  difference  tended  to  be  greater  at  higher  levels  of  current  level  of  conceptual  capacity.  The 
inter-rater  reliability  was  .95  for  two  raters.  The  inter-rater  reliability  for  our  CGSOC  sample  in  this  study 
was  .88. 


Biodata.  Biodata  is  a  scale  developed  by  ARI.  The  instrument  consists  of  87  questions  that 
generally  ask  how  often  or  to  what  extent  an  individual  can  be  described  in  a  certain  way.  The  test  results 
in  scores  on  eight  scales:  achievement  motivation,  openness  to  new  ideas,  dominance,  peer  leadership, 
cognitive  complexity,  need  for  personal  approval  from  others,  orientation  towards  others  as  objects  (a 
depersonalizing  orientation),  and  a  lie  scale.  Biodata  refers  to  biographical  information  that  describes  an 
individual's  past  experiences.  Typical  biodata  tests  use  instances  that  can  be  easily  and  objectively 
reported,  (e.g.,  How  many  books  do  you  read  a  month?  What  is  this  individual’s  grade  point  average?) 
From  these  objective  accounts,  inferences  can  be  made  about  an  individual.  The  data  are  not  necessarily 
obtained  only  from  self-reports  but  from  existing  records  or  other  people.  Instead  of  relying  on  specific 
instances,  the  ARI-developed  Biodata  scales  included  self-reports  of  a  more  general  nature.  (For  example 
to  the  question  "How  often  have  you  been  recognized  for  unusually  effective  work  performance?"  the 
response  choices  were  "very  often,"  "often,"  "sometimes,"  "seldom,"  and  "never.")  Biodata  was  taken  from 
40  MSF  and  27  II  Corps  officers. 


Azimuth  Azimuth  is  an  instrument  which  intends  to  aid  self-development  of  leadership  skills. 
Items  reflecting  important  qualities  of  leadership  are  completed  by  the  person  on  him  or  herself  and  on 
peers.  Feedback  is  provided  based  on  a  comparison  of  individuals'  self-perceptions  and  one's  peer's 
perceptions  of  them.  Items  belong  to  one  of  12  elements  generally  corresponding  to  doctrinal  leadership 
competencies  (Headquarters,  1990). 
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The  twelve  elements  calculated  from  Azimuth  items  are  described  as  follows. 


Supervisory  Skills.  This  broad  category  of  leader  behavior  is  broken  out  into  the  following  three 
elements: 

Interpersonal  focus:  This  element  examines  the  way  in  which  a  leader  interacts  with 
subordinates.  It  is  concerned  with  both  interpersonal  behavior  and  empowerment  of 
subordinates  to  make  decisions  and  grow. 

Team  focus:  This  element  looks  at  leadership  in  developing,  motivating,  and  resourcing 
teams.  It  is  concerned  with  ability  to  select  and  challenge  a  high  performing  team. 

Mission  Focus:  Mission  focus  looks  at  the  leadership  provided  to  create  a  supportive 
environment  in  which  subordinates  can  accomplish  their  mission. 

Tactical  and  Technical  Competence.  Tactical  and  technical  competence  is  also  broken  down  into 
three  specific  elements: 

Problem  solving  skills:  This  element  looks  at  the  leader  as  a  decision  maker  and  the  ability 
to  "get  to  the  heart  of  the  matter"  and  to  sort  out  the  important  from  the  unimportant 
details. 

Knowledge:  The  knowledge  element  looks  at  the  degree  to  which  the  leader  is  an 
accomplished  professional  in  a  technical  or  tactical  area  of  expertise. 

Planning/Organizational  skills:  The  planning  and  organizational  skills  element  addresses 
whether  the  leader  can  meet  deadlines,  can  maintain  focus  on  primary  issues  even  when 
other  things  compete  for  attention,  and  can  see  the  "big  picture". 

Political  Skills.  This  element  is  concerned  with  sensitivity  to  political  issues  and  interests  beyond 
the  purely  military.  This  element  includes  economic  and  societal  interests,  and  the  ability  to 
maintain  good  relationships  with  outside  interest  groups,  as  well  as  peers  and  superiors. 

Ethics.  This  element  is  concerned  with  the  leader's  personal  and  professional  standard  of  conduct, 
and  includes  a  willingness  to  be  accountable  for  personal  failures  and  to  set  the  proper  example  of 
high  standards  of  interpersonal  responsibility  and  honesty. 

Communication/Influence.  Leadership  requires  the  ability  to  establish  and  maintain  effective 
two-way  communication  with  others,  including  peers,  subordinates  and  superiors.  This  includes 
the  establishment  and  maintenance  of  both  formal  and  informal  channels  of  communication  and 
information  dissemination,  as  well  as  the  ability  to  listen  effectively  and  provide  an  atmosphere 
conducive  to  openness  and  honesty. 

Social  Maturity.  This  element  is  concerned  with  the  ability  to  maintain  composure  under 
conditions  which  may  be  personally  threatening,  the  ability  to  admit  and  learn  from  your  mistakes, 
and  willingness  to  consider  the  opinions  of  others  which  may  differ  from  the  leader's  own  opinions. 
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Self-Centeredness.  This  is  a  negative  element  in  that  the  questions  which  define  it  concern 
behaviors  to  be  avoided.  These  behaviors  indicate  poor  leadership.  This  element  is  centered 
around  the  focus  on  self  and  lack  of  concern  and  care  for  others.  This  element  describes  an 
individual  who  is  arrogant,  vindictive,  impressed  with  self,  and  has  an  explosive  temper. 

Compulsive  Behavior.  Like  the  element  above,  this  element  is  concerned  with  behaviors  which 
may  be  indicators  of  poor  leadership.  This  element  examines  the  tendency  to  micromanage,  to 
work  self  and  others  to  excess,  be  intolerant  of  ambiguity,  and  to  desire  excessive  information 
before  making  decisions. 

Additional  Items  Relating  to  Prairie  Warrior.  Twenty-two  supplemental  questions  were  added  to 
the  current  set  of  98  Azimuth  items.  Six  of  these  mirrored  6  of  the  7  critical  information 
discernment  behaviors  that  were  rated  by  observers  in  the  exercise.  Other  questions  dealt  with 
problem  solving  and  reasoning  behaviors,  general  problem  solving  styles,  use  and  knowledge  of 
systems,  and  confidence  and  generalizability  of  the  ratings. 

Spearman  Brown  reliability  coefficients  were  computed  on  peer  ratings  from  a  sample  of  545 
CAS3  students.  Reliability  coefficients  ranged  from  .69  to  .78  for  all  but  the  compulsive  behavior  element, 
which  was  .48. 

Admini  stration  of  Azimuth.  Azimuth  was  administered  to  66  students  who  rated  themselves  and 
peers.  Students  were  assigned  to  rate  two  or  three  peers.  The  assignments  were  designed  to  include  a 
student  who  was  in  the  same  cell  as  the  ratee,  one  student  who  may  have  principally  provided  information 
to  the  ratee,  and  one  student  who  may  have  received  information  products  from  the  ratee.  Subordinate 
commanders  were  rated  by  a  commander,  deputy  commander,  chief  of  staff,  or  operations  officer.  Once 
these  criteria  were  met,  raters  were  selected  who  had  branch  commonality  with  the  students  they  were  to 
rate.  For  15  students  one  of  their  ratings  came  from  1 1  senior  noncommissioned  officers  from  the  US 
Army  Sergeants  Major  Academywho  were  also  participating  in  the  PW96  exercise.  Ratings  of  a  student 
were  combined  into  a  category  of  ratings  from  peers,  regardless  of  whether  the  raters  were  playing  the  role 
of  a  superior,  subordinate,  or  peer. 


Critical  Information  Discernment 

Critical  Information  Discernment  Behaviors.  One  of  the  requirements  faced  in  this  study  was  to 
define  critical  information  discernment  so  that  it  could  be  measured.  Three  different  categories  were 
defined  to  differentiate  among  behaviors  (see  Table  1).  The  general  categories  were:  information  handling, 
information  usage,  and  integration. 

Information  handling  consisted  of  three  subcategories:  Commander's  critical  information 
requirements  (CCIR),  filtering  information,  and  seeking  information. 

CCIR  is  a  formalization  of  information  handling  in  which  the  staff  identifies,  plans, 
collects,  and  manages  critical  information  elements.  There  is  no  standard  set  of  information 
elements  that  are  universally  applicable,  so  doctrine  establishes  a  formal  policy  for  the  processing 
of  CCIR.  The  CCIR  process  may  be  an  overspecialization,  because  the  process  is  not  always 
followed  closely.  Often  CCIR  are  considered  prior  to  a  mission  or  exercise  and  not  kept  current  or 
updated.  Once  a  mission  or  exercise  starts,  attention  is  directed  at  specific  information  and  less  on 
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Table  1. 

Critical  Information  Discernment  behaviors. 


Category 

Behavior 

Information  handling 

Management  and  attention  to  formal  CCIR. 

Filtering  of  information. 

Information  sought  out  proactively. 

Use  of  information 

Specific  predictions  of  battlefield  situations. 

Kept  track  of  minor-but  potentially  critical  situations-as  they  developed. 

Identified  or  anticipated  major  changes  in  enemy  or  friendly  intentions, 
schedules,  resources,  and  boundaries. 

Integration 

Integrated  information  to  produce  critical  insights. 

the  management  of  the  information.  There  is  a  danger  of  an  exclusive  focus  on  one  or  the  other 
tendency.  If  a  staff  or  an  individual  focuses  on  the  process  of  information  management,  they  may 
not  understand  as  well  as  they  should  the  meaning  of  the  information,  or  if  they  focus  on  what 
specific  information  means  they  may  lose  sight  of  other  information  or  sources  that  are  available 
and  important. 

Information  handling  also  involves  the  active  search  for  information.  If  information  is  not 
actively  sought  out,  the  staff  will  not  have  the  best  and  latest  information  with  which  to  work. 
New— and  especially  unexpected  information— undoubtedly  suggests  other  possibilities  of  what  is 
happening  on  the  battlefield.  These  other  possibilities  are  cues  to  be  confirmed  or  disconfirmed 
through  additional  information  search.  Confirming  and  clarifying  the  situation  should  be  a 
continual  process. 

Filtering  of  information  is  an  important  part  of  information  handling.  Whether  information 
has  been  received  without  request  or  has  been  actively  sought,  the  sheer  magnitude  requires  that  it 
be  reduced  to  that  which  is  most  important  and  most  time-critical.  The  most  critical  is  not  always 
that  which  is  most  immediate— has  the  most  sudden  implications.  If  the  staff  focuses  too  greatly  on 
routine  handling  of  all  information,  the  less  important  information  will  bog  down  processing  and 
lead  to  the  lack  of  clarity  in  battlefield  visualization. 

Information  usage  was  separated  into  three  categories.  The  categories  were  defined  to  be  general 
enough  to  include  battlefield  visualization  and  decision  making,  regardless  of  echelon  or  functional  area. 
The  subcategories  are:  prediction,  tracking  of  minor  situations,  and  identification  of  major  changes. 

Information  is  used  to  serve  as  a  basis  for  prediction.  Predictions  of  battlefield  situations 
and  outcomes  are  important  for  preparing  for  future  operations.  Predictions  should  be  based  on 
the  best  information  available. 

Tracking  minor  situations  is  an  important  task  that  also  is  based  on  information  received. 
Since  dynamic  battlefield  situations  are  full  of  uncertainty  and  intentional  deception,  what  might 
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seem  to  be  a  minor  and  trivial  event  might  actually  be  the  only  warning  of  a  significant  enemy 
action  or  a  friendly  difficulty  to  perform.  What  might  appear  to  be  the  sign  or  cue  of  one  event 
might  actually  be  indicating  something  quite  different.  Interpretations  that  keep  multiple 
possibilities  open  is  one  way  of  dealing  with  battlefield  uncertainty. 

Anticipation  or  identification  of  major  changes  in  plans  or  operations  is  another  key  task  of 
Critical  Information  discernment.  Changes  can  occur  about  intentions,  schedules,  resources,  and 
boundaries.  No  operation  goes  completely  as  planned  or  can  be  entirely  envisioned.  The  more 
successful  commander  will  have  a  clear  understanding  of  the  current  situation,  friendly  plan, 
enemy  capabilities,  and  some  idea  about  the  future. 

Observers.  Observers  came  from  five  different  organizations.  All  had  work-related  experience  in 
the  study  and  observation  of  command  and  staff  behavior.  Fifteen  of  the  observers  came  from  either  of  two 
Battle  Command  Training  Program  teams.  These  observers'  day-to-day  jobs  require  that  they  watch  units 
performing  battle  staff  exercises  and  provide  training  feedback  to  them.  Twelve  other  observers  were 
instructors  from  the  Center  for  Army  Tactics,  where  they  instruct  CGSOC  students  and  observe 
performance  in  classroom  exercises.  The  other  five  observers  came  from  ARI,  CAL,  and  the  TRADOC 
Analysis  Center  (TRAC).  These  observers  had  extensive  experience  observing  command  and  staff 
exercises  during  BCTP  Warfighter  Exercises,  CTC  rotations,  and  in  the  classroom. 

Ratings  of  Behaviors.  A  Critical  Information  discernment  rating  score  sheet  was  used  by  the 
observers  to  provide  ratings  and  explanations  of  events  that  they  observed  (see  Appendix  A).  An  earlier 
version  of  the  score  sheet  was  tried  out  during  a  previous  simulation  exercise  for  the  MSF  students. 

Several  of  the  observers  participated  and  provided  feedback  on  the  sheets  and  rating  procedures. 

During  PW96,  observers  were  asked  to  rate  each  student  they  observed  daily  using  the  score  sheet. 

Frequency  ratings  were  to  show  when  the  Critical  Information  discernment  behaviors  were  not  warranted; 
observers  were  instructed  to  rate  only  those  behaviors  observed. 

Observers  were  also  prompted  to  remark  about  extenuating  circumstances  for  the  behaviors.  They 
were  asked  to  comment  about  whether  the  student  was  performing  at  or  below  potential,  whether  the 
student  was  constrained  by  position  or  command  relationship,  whether  the  realism  or  quality  of  information 
was  suspect,  and  how  much  of  a  challenge  the  opposing  force  was. 

Processing  of  Ratings.  The  frequency  and  quality  ratings  given  by  observers  were  combined  into  a 
single  rating  for  each  Critical  Information  discernment  behavior.  Table  2  shows  how  the  separate  ratings 
were  combined  into  a  single  rating.  For  example,  if  a  student  received  a  frequency  rating  of  Usually  and  a 
quality  rating  of  8,  a  combined  rating  from  the  look-up  table  would  have  been  a  9.  The  quality  and 
combined  ratings  correlated  at  .90  suggesting  that  the  combination  and  frequency  did  not  affect  the  results. 

Panel  Ratings  Three  of  the  experimenters  reviewed  the  observation  sheets  independent  of  one 
another  and  made  an  initial  recommendation  to  categorize  the  reported  behaviors  and  ratings  according  to 
low,  medium,  or  high  performance  categories.  There  were  33  matches  in  assignment  and  17  mismatches. 
Mismatches  were  no  further  apart  than  a  neighboring  category.  The  panel  then  met  for  four  hours  to 
discuss  the  assignments  and  to  resolve  any  differences.  Panel  members  resolved  differences  by  judging  the 
quality  of  explanations  or  from  observations  that  one  of  the  members  had  made  to  confirm  or  disconfirm 
an  observer's  rating.  A  fourth  CID  category  of  medium-high  was  added  to  more  accurately  differentiate 
among  students'  behaviors. 
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Table  2. 

Combination  of  Frequency  and  Quality  Ratings  of  Information  Discernment 

Quality 


10  9 

8  7 

6  5 

4 

3 

2  1 

Always 

10 

9 

8 

5 

4 

Usually 

Seldom 

7 

5 

3 

Never 

2 

1 

Two  of  the  panel  members  independently  ranked  the  students  in  the  high  and  medium  high 
categories  and  again  met  to  resolve  discrepancies.  Since  one  of  the  prime  reasons  for  low  categorizations 
was  the  absence  of  explanations  on  CID  instances,  there  was  little  qualitative  interpretation  that  could  be 
done  in  the  two  lowest  categories.  For  the  low  and  medium  categories  the  average  combined  rating  for 
behaviors  4,  5  and  7  was  used  to  rank  order  these  students  with  behaviors  1,  2  and  3  breaking  ties. 
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Results 


Overview 

There  was  no  direct  relationship  between  the  modified  CPA  (MCPA)  measure  of  Conceptual 
Capacity  Level  and  ranking  of  CID  performance  (r=  -.07)  (see  Figure  1).  There  were  moderate  negative 
correlations  between  CID  performance  and  the  Biodata's  Cognitive  Complexity  (r=  -.25,  p=  .09)  and 
Openness  scales  (r=  -.28,  p=  .05). 


Figure  1.  Critical  information  discernment  and  conceptual  capacity  level  (r=  -.07). 


Critical  Information  Discernment 

Quantity  of  ratings.  The  number  of  ratings  differed  among  the  seven  behaviors  and  among 
students  being  rated.  For  the  seven  behaviors,  the  fewest  number  of  ratings  was  178  and  the  most  was  227 
(aggregating  across  the  seven  days  of  the  exercise  and  across  individuals).  About  40  percent  of  the 
students'  behavior  was  based  on  ratings  from  six  to  eight  days  of  observation.  Another  36  percent  was 
based  on  four  or  five  days,  and  the  remaining  24  percent  were  based  on  one,  two,  or  three  days  of 
observation. 

Explanations  of  ratings.  A  sample  of  seven  of  the  32  observers'  ratings  was  examined  to 
understand  (1)  whether  direct  or  indirect  relationships  to  the  behavior  were  made  by  the  raters  and 
(2)  whether  specific  or  general  support  was  given  in  their  explanations.  These  issues  relate  to  the  adequacy 
of  measurement  methods,  the  quality  of  raters,  and  the  behaviors  displayed  by  the  students  as  determined 
by  a  subsample  of  17  students.  Over  97  percent  of  the  explanations  were  judged  to  be  supportive  of  the 
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specific  behavior  being  rated.  Over  70  percent  of  the  explanations  for  prediction,  tracking,  and  identifying 
major  changes,  referred  to  specific  instances  dealing  with  tactics  and  about  one  fifth  dealt  with  more 
general  behavior.  For  the  integration  behavior,  about  two  fifths  of  the  observations  cited  specific  instances 
and  three  fifths  cited  general  tendencies. 

Over  70  percent  of  the  explanations  on  tracking  minor  issues  had  direct  relationships  to  the 
behavior.  Explanations  on  identification  of  major  changes,  had  a  reverse  trend.  Seventy-two  percent  were 
indirectly  related.  (An  indirect  explanation  suggests  that  the  observer  was  considering  multiple  instances  of 
the  behavior  or  interpreting  the  item  more  broadly  than  the  method  intended.)  Explanations  for  integration 
and  prediction  showed  about  35  percent  of  the  relationships  to  be  directly  related  to  these  two  behavior 
categories. 

Type  of  job.  A  multiple  regression  between  type  of  job  and  performance  showed  no  significant 
relationship  (r2=  .09,  F(3>46)=  1.44).  Types  of  jobs  considered  included  command,  operations,  intelligence, 
and  service  support.  This  finding  implies  that  students  did  not  perform  differently  at  Critical  Information 
discernment  across  the  different  command  and  staff  positions. 

Motivation.  Estimates  of  student  motivation  were  made  on  25  students.  These  estimates  were 
made  from  the  interpretation  of  an  observer's  comments  about  the  student's  performance  potential  and 
morale,  independent  of  the  CID  performance  ratings.  A  significant  correlation  was  found  between 
discernment  ranking  and  motivation  (r=  .487,  p=  .01).  There  was  a  near-zero  correlation  between 
discernment  and  Conceptual  Capacity  Level  when  motivation  was  partialled  out  (rc  ci  m=  .07). 

Time  pressure.  An  estimate  of  the  amount  of  challenge  that  the  students  were  under  came  from 
item  1 15  on  the  Azimuth  instrument.  When  the  students’  ratings  of  time  pressure  were  removed,  the 
correlation  between  discernment  and  Conceptual  Capacity  Level  was  still  near-zero  (rc>ci  t=  -01). 

Organizational  differences.  The  Student  Corps  performed  better  than  the  MSF  Division  as 
determined  by  comparison  of  the  distribution  of  scores  across  the  five  performance  categories  {Flt48  =  4.55, 
P<- 05) 


Conceptual  Capacity 

There  was  no  significant  difference  in  the  distribution  of  Conceptual  Capacity  Level  scores  for  this 
sample  of  65  CGSOC  students  and  those  of  higher  rank  from  the  earlier  Army  War  College  (AWC)  sample 
(Lewis,  1993)  (see  Figure  2)  (x2is=  13.07,  p=  .60). 


Biodata  correlations  with  Critical  Information  Discernment.  The  Biodata  scales  did  not  have  very 
large  correlations  with  the  performance  criterion.  Only  the  Openness  scale  had  a  significant  negative 
correlation  with  ranking  of  Critical  Information  discernment  (see  Table  3). 
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Figure  2.  Cumulative  distribution  of  conceptual  capacity  level  for  CGSOC  and  AWC 


Table  3.  Correlations  and  Observed  Significance  Levels  between  CID  Ranking  and  Biodata  Scales  (n=49, 
correlation  in  bold,  significance  level  below) 


Achievment 

Openness 

Dominance 

Peer 

Leadership 

Cognitive 

Complexity 

Need  for 
Approval 

Object 

Orientation 

Lie 

Ranking  of 

.15 

-.28 

.07 

-.13 

-.25 

.16 

.25 

-.01 

CID 

.30 

.05 

.63 

.36 

.09 

.27 

.09 

.96 

Additional  survey  questions.  After  an  initial  look  at  the  results  and  the  lack  of  a  relationship 
between  Conceptual  Capacity  Level  and  performance,  available  students  were  surveyed.  They  were  asked 
four  additional  questions  dealing  with  their  perceptions  about  the  challenges  of  the  exercise.  The  intent  of 
these  questions  were  to  address  whether  the  load  of  the  exercise  was  so  low  that  any  relationship  was 
masked.  Responses  were  received  from  28  of  the  MSF  students  and  17  of  the  II  Corps  students. 

The  four  items  were: 

1 .  Rate  the  degree  of  task  difficulty  you  expected  during  PW96.  (7  very  difficult  to  1  not 
difficult) 

2.  Rate  the  degree  of  task  difficulty  you  experienced  during  PW96. 

3.  If  there  was  a  difference  in  what  you  expected  and  what  you  experienced,  explain  why. 

4.  To  what  degree  of  similarity  was  the  job  you  performed  in  PW96  compared  to  what  you 
have  done  in  the  past?  (7  great  deal  of  similarity  to  1  no  similarity) 
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Difficulty.  Thirty-six  (80%)  of  the  students  reported  that  experienced  difficulty  was  medium  or 
higher.  Twenty  (44.4%)  of  the  student's  experienced  difficulty  matched  what  they  expected.  Eighteen 
(40%)  of  the  student's  experienced  difficulty  was  lower  than  they  expected,  and  seven  (15.6%)  had 
experienced  difficulty  that  was  higher  than  expected. 

Twenty-five  students  provided  an  explanation  for  their  change  in  assessment  of  difficulty.  Some 
(18  of  25)  found  difficulty  to  be  less  than  expected.  Seven  of  18  discussed  this  as  a  positive  change  and  11 
felt  negatively  about  the  change  (see  Tables  4  and  5).  Five  of  the  25  students  providing  an  explanation 
reported  that  their  job  was  not  challenging. 

Table  4. 

Change  from  expected  to  experienced  difficulty  and  interpretation  of  change. 


Change  in  Difficulty 

Pleased 

Displeased 

Totals 

Lower  than  expected 

7 

11 

18 

Higher  than  expected 

3 

4 

7 

Totals 

10 

15 

25 

Table  5. 

Reasons  for  changes  in  perception  of  difficulty. 


Student  Explanations  for  Change  in  Perception 

Frequency 

1  Positive  j| 

Fellow  students  were  better  than  anticipated. 

2 

The  job  requirements  became  clearer  and  work  was  more  satisfying. 

2 

Own  anxiety  was  not  as  high  as  expected. 

2 

The  units'  plans  worked  out  better  than  expected. 

2 

The  simulation  supported  learning  better  than  expected. 

i 

Expected  that  working  at  a  higher  echelon  would  be  more  difficult. 

1 

Negative 

Job  was  not  challenging. 

5 

Systems  (e.g.,  logistics,  aviation,  intelligence)  were  not  played  adequately. 

5 

Fellow  students  were  not  as  involved  as  they  ought  to  have  been. 

2 

Doctrine  and  tactics  were  more  difficult  than  expected. 

1 

Expectations  on  students  from  outside  the  units  increased. 

1 

Trainers  caused  distractions. 

1 
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There  was  a  positive  relationship  between  difficulty  and  performance;  the  higher  that  experienced 
difficulty  was  rated  the  better  Critical  Information  discernment  was  (rdi$cr=  .38,  p=  .03).  Experienced 
difficulty  had  no  significant  correlation  with  Conceptual  Capacity  Level  or  the  Biodata  Openness  or 
Cognitive  Complexity  scores.  Partialling  out  experience  did  not  noticeably  increase  the  relationship 
between  Conceptual  Capacity  Level  and  Critical  Information  discernment  (rciD.cL.exp  =  .02,  p  =  .93). 

Similarity.  Forty-two  of  the  students  rated  similarity  of  their  PW96  job  to  what  they  had  done  in 
the  past.  The  average  rating  was  4. 1 .  Rated  similarity  did  not  correlate  with  Critical  Information 
discernment  (rsimjcr=  -  03),  Conceptual  Capacity  Level  (rsim,d=  .16),  or  Cognitive  Complexity  (rsim,Cc=  17). 
Rated  similarity  had  a  positive  correlation  with  the  Biodata  Openness  score  (rSim>op=:  .398,  p=.009).  This 
relationship  suggests  that  those  rating  similarity  higher  were  more  open  to  seeing  an  application  of  their 
previous  experience.  There  was  no  correlation  between  Conceptual  Capacity  Level  and  Critical 
Information  discernment  when  similarity  was  partialled  out.  The  negative  correlation  between  Openness 
and  Critical  Information  discernment  increased  in  magnitude  when  similarity  was  partialled  out  (rop,cr.sim=  - 
.37,  p=.05). 

Challenge  of  the  Task.  The  magnitude  of  the  negative  correlation  between  Openness  and  Critical 
Information  discernment  increased  when  the  effects  of  several  task  complexity  variables  were 
simultaneously  partialled  out.  These  task  complexity  variables  were: 

The  rank  of  the  position  of  a  student's  job. 

Experienced  difficulty. 

Expected  difficulty. 

Similarity  of  experience  to  their  job. 

Self-assessment  of  the  time  pressure  they  were  under. 

The  partial  correlation  between  Openness  and  Critical  Information  discernment  was  negative  (  rCiD,open  chau= 
-.43,  p=  .03).  The  partial  correlation  between  Conceptual  Capacity  Level  and  Critical  Information 
discernment  was  still  near  zero  (  rcro  ,CL.Chall •  039). 

Students  did  not  indicate  a  lack  of  interest  or  challenge  (80%  of  them  reported  experienced 
difficulty  equal  to  or  greater  than  die  midpoint  of  the  seven  point  scale  and  only  10  percent  attributed 
displeasure  to  lack  of  a  challenging  job).  Nor  did  they  indicate  that  their  jobs  in  this  exercise  were  very 
similar  to  their  experience  (half  of  them  rated  similarity  at  the  midpoint  or  higher).  However,  if  task 
complexity  was  controlled  for,  the  relationship  between  Openness  and  Critical  Information  discernment 
became  stronger  in  the  negative  direction. 


Biodata  Correlations  with  Current  Level.  While  the  largest  correlation  of  note  so  far  showed  a 
negative  relationship  between  Openness  and  Critical  Information  discernment  rank,  there  were  significant 
correlations  between  Current  Level  of  Conceptual  Capacity  and  four  of  the  Biodata  scales,  (see  Table  6). 
Openness  and  Cognitive  Complexity  had  the  largest  correlations  with  Conceptual  Capacity  Level,  while 
Dominance  and  Need  for  Personal  Approval  from  Others  had  more  modest  correlations  with  Conceptual 
Capacity  Level.  The  three  other  scales  did  not  have  significant  correlations  with  Conceptual  Capacity 
Level. 
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Table  6.  Correlations  and  Observed  Significance  Levels  between  Current  Level  of  Conceptual  Capacity 
and  Biodata  Scales  (n=50,  correlation  in  bold,  significance  level  below) 


Achievment 

Dominance 

Peer 

Leadership 

Cognitive 

Complexity 

Need  for 
Approval 

Object 

Orientation 

Lie 

.18 

.35 

.17 

■■ESI 

.29 

.14 

-.12 

.23 

.01 

.25 

.05 

.35 

.40 

Azimuth  Results 

Azimuth  Correlations  with  Performance.  Azimuth  provides  an  alternate  set  of  predictors  for  the 
performance  measure  of  Critical  Information  discernment  and  the  performance  measure  provides  an 
independent  criterion  for  gauging  Azimuth  results.  There  were  no  significant  correlations  between 
performance  and  elements  rated  by  peers  (correlations  ranged  from  rCT,EL7  =  -.  1 6  to  rcr,ELi2  =  .  12).  There 
were  no  significant  correlations  between  performance  and  self-rated  elements  (correlations  ranged  from 
I*cr,ELS7  =  "•  27  to  r^ELsio  =  -.09)  (see  Table  7). 

Table  7.  Pearson  product-moment  correlations  between  ranking  of  Critical  Information  Discernment  and 
Azimuth  element  ratings  (sample  size  =  49  to  50). 


Azimuth  Elements 

Self  ratings 

Peer  ratings 

Interpersonal  focus 

-.21 

-.02 

Team  focus 

-.20 

-.03 

Mission  focus 

-.14 

-.09 

Problem  solving  skills 

-.21 

.01 

Knowledge 

-.14 

.03 

Planning/organizational  skills 

-.11 

-.04 

Political  skills 

-.27 

-.16 

Ethics 

-.19 

.00 

Communication/influence 

-.12 

-.02 

Social  maturity 

-.10 

-.05 

Self-centeredness 

-.22 

-.02 

Compulsive  behavior 

-.22 

.12 

Only  five  of  the  individual  self-rated  items  correlated  significantly  with  the  performance  criterion 
(see  Table  8).  These  all  had  negative  correlations,  indicating  that  the  students  ranking  high  on  performance 
thought  these  items  were  not  as  descriptive  of  themselves  compared  to  the  students  who  performed  poorly. 
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Table  8. 

Azimuth  items  correlating  with  CID  (*  <  .05,  **  <  .01) 


Azimuth  Item 

Correlation 
with  CID 

60.  [Does  not]  micromanage. 

-.37  ** 

71 .  Seeks  knowledge  about  world  political  and  economic  conditions. 

-.40  ** 

76.  Shows  interest  in  the  professional  growth  of  subordinates. 

_  35  ** 

94.  Uses  praise  to  accomplish  the  organization's  objectives. 

-.31  * 

98.  Poes  not]  want  it  done  "my  way"  or  no  way. 

-.33  * 

Besides  those  5  items  mentioned  above,  several  of  the  self-rated  items  had  significant  negative 
correlations  with  the  individual  discernment  behaviors  rated  by  observers  (see  Table  9). 

Table  9. 

Self-rated  items  correlating  negatively  with  one  or  more  of  the  information  discernment  behaviors. 


Azimuth  Item 

Info. 

Reqts 

Filter 

Search 

Id. 

change 

8.  Quick  decisions  when  circumstances  call  for  them. 

-.36 

-.28 

-.32 

-.32 

31 .  poes  not]  insist  on  precision  in  trivial  matters. 

-.30 

53.  Knows  policy  or  doctrine. 

-.33 

-.30 

-.33 

-.33 

55.  poes  not]  lack  sufficient  technical  competence. 

-.30 

-.30 

67.  Persuasive. 

-.28 

-.31 

69.  Provides  coordination  for  subordinate  organizations  ... 

-.37 

-.30 

-.31 

73.  Sees  the  pattern  in  seemingly  unrelated  problems. 

-.31 

-.31 

78.  Stays  composed  when  under  personal  attacks  by  others. 

-.34 

-.31 

-.32 

90.  Understands  an  issue  before  making  a  decision. 

-.36 

-.33 

-.31 

The  negative  correlations  indicate  disagreement  between  self-perception  of  how  good  one  is  at  problem¬ 
solving  processes  and  their  performance  as  rated  by  peers. 

Only  one  of  the  items  rated  by  peers  correlated  significantly  with  the  performance  criterion.  The 
item  was  #62,  "Moves  quickly  to  confront  problem  subordinates."  and  the  correlation  was  -.31  (p<.05). 

Besides  this  item,  five  more  individual  items  rated  by  peers  had  significant  negative  correlations 
with  CID  rankings  (see  Table  10). 
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Table  10. 

Azimuth  items  rated  by  peers  correlating  with  CID  (*  <  .05,  **  <  .01) 


Azimuth  Item 

Correlation 
with  CID 

12.  [Does  not]  criticize  others  in  front  of  others. 

-.30* 

14.  Emphasizes  subordinates  strengths. 

-.29  * 

29.  His  or  her  plans  [do  not]  need  frequent  revision. 

-.32  * 

40.  Is  [not]  impressed  with  his  or  her  own  rank  and  status. 

-.28* 

65.  [Does  not]  often  lose  his  or  her  temper. 

-.44  ** 

The  peer  and  self-rated  elements  were  regressed  on  the  performance  criterion.  Stepwise  multiple 
regression  resulted  in  a  low  proportion  of  shared  variance  (r2=  .  19,  F  =  2.60,  p=  .049)  with  four  elements 
giving  the  best  prediction:  self-rated  planning  and  organizational  skills  (weight  of  4.21);  peer-rated 
compulsive  behavior  (weight  of  5.98);  self-rated  and  peer-rated  political  skills  (negative  weights  of  9.37 
and  13.27,  respectively).  The  pattern  of  correlations  and  multiple  regressions  of  Azimuth  elements  and 
items  showed  no  strong  relationships  to  Critical  Information  discernment. 

Azimuth  Correlations  with  Conceptual  Capacity  Level.  Three  of  the  12  self  Azimuth  element 
scores  (problem  solving  skills,  knowledge  and  political  skills)  correlated  significantly  with  Conceptual 
Capacity  Level.  All  of  the  element  scores  from  peer  ratings  had  negative  correlations  with  Conceptual 
Capacity  Level.  The  six  below  -.24  were  significant  (see  Table  1 1) 

Table  1 1 .  Pearson  product-moment  correlations  between  ranking  of  Conceptual  Capacity  Level  and 
Azimuth  element  ratings  (sample  size  =  65). 


Azimuth  Elements 

Self  ratings 

Peer  ratings 

Interpersonal  focus 

.04 

-.36*  li 

Team  focus 

.12 

-.42* 

Mission  focus 

.17 

-.24 

Problem  solving  skills 

.36* 

-.25* 

Knowledge 

.27* 

-.23 

Planning/organizational  skills 

.18 

-.20  : 

Political  skills 

.36* 

-.20 

Ethics 

.08 

-.25* 

Communication/influence 

.13 

-.32* 

Social  maturity 

.07 

-.14 

Self-centeredness 

-.03 

-.23 

Compulsive  behavior 

.13 

-.30* 
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Correlations  between  the  element  scores  from  self  and  the  Openness  and  Cognitive  Complexity 
(CC)  subtests  of  the  Biodata  instrument  followed  similar  patterns  as  Conceptual  Capacity  Level  (see  Table 
12  &  13).  All  but  Ethics  and  Compulsiveness  were  significantly  correlated  with  the  Openness  factor.  The 
elements  of  Mission  Focus,  Political  Skills,  and  Communication  were  all  correlated  with  the  Cognitive 
Complexity  factor.  Correlations  between  the  element  scores  from  peers  and  the  Openness  and  Cognitive 
Complexity  factors  were  not  significant,  but  tended  to  go  in  a  negative  direction. 

Table  12.  Pearson  product-moment  correlations  between  ranking  of  Biodata's  Openness  and  Azimuth 
element  ratings  (sample  size  =  67). 


Azimuth  Elements 

Self  ratings 

Peers'  ratings 

Interpersonal  focus 

.34* 

-.17 

Team  focus 

.38* 

-.23 

Mission  focus 

.45* 

-.16 

Problem  solving  skills 

.48* 

-.12 

Knowledge 

.44* 

-.09 

Planning/organizational  skills 

.50* 

-.03 

Political  skills 

.57* 

-.02 

Ethics 

.18 

-.14 

Communication/influence 

.42* 

-.18 

Social  maturity 

.35* 

-.02 

Self-centeredness 

.25* 

-.10 

Compulsive  behavior 

.20 

-.12 

Thirty-seven  of  the  peer-rated  items  correlated  significantly  negatively  with  Conceptual  Capacity 
Level.  Forty-eight  of  the  self-rated  items  correlated  positively  with  Conceptual  Capacity  Level.  Only  two 
were  the  same  item  between  the  37  "peer"  items  and  48  "self1  items.  These  two  were  item  #67, 
"Persuasive. "  and  item  #69,  "Provides  the  coordination  for  subordinate  organizations  to  accomplish 
interrelated  tasks. "  The  self-rated  items  both  correlated  with  Conceptual  Capacity  Level  at  .28.  The 
'peer'  rated  items  correlated  negatively  at  -.35  and  -.29,  respectively. 


Azimuth  Supplemental.  None  of  the  self-ratings  (see  Table  14)  or  peer-ratings  (see  Table  15)  for 
the  six  supplemental  items  on  information  discernment  behaviors  correlated  with  the  same  items  rated  by 
observers.  Also  there  were  no  significant  correlations  on  these  six  items  between  the  students'  self-ratings 
and  the  ratings  by  peers  (Table  16).  The  ratings  of  Critical  Information  discernment  behaviors  by 
observers,  student  peers,  and  self  did  not  correlate  with  each  other. 
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Table  13.  Pearson  product-moment  correlations  between  ranking  of  Biodata's  Cognitive  Complexity  and 
Azimuth  element  ratings  (sample  size  =  65). 


Azimuth  Elements 

Self  ratings 

Peers'  ratings 

Interpersonal  focus 

.14 

-.11 

Team  focus  ; 

.10 

-.17 

Mission  focus 

.25* 

-.09 

Problem  solving  skills 

.20 

-.06 

Knowledge 

.22 

-.10 

Planning/organizational  skills 

.23 

-.07 

Political  skills 

.44* 

-.05 

Ethics 

.06 

-.19 

Communication/influence 

.26* 

-.14 

Social  maturity 

.16 

-.05 

Self-centeredness 

.19 

-.03 

Compulsive  behavior 

.04 

-.12 

Table  14.  Comparison  of  self-ratings  and  observer  ratings  on  information  discernment  behaviors. 


Self  ratings  of  performance  behaviors 


CCJR 

Filtering 

Search 

Integration 

Prediction 

Tracking 

CCIR 

-.05 

Observer 

Filtering 

-.03 

ratings  of 

Search 

-.10 

performance 

Integration 

-.15 

Prediction 

-.15 

Tracking 

-.09 
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Table  15.  Comparison  of  peer  ratings  and  observer  ratings  on  information  discernment  behaviors. 


Peer  ratings  of  performance  behaviors 


CCIR 

Filtering 

Search 

Integration 

Prediction 

Tracking 

CCIR 

.00 

Observer 

Filtering 

.06 

ratings  of 

Search 

.03 

performance 

Integration 

-.20 

Prediction 

-.05 

Tracking 

.01 

Table  16.  Comparison  of  Azimuth  self  ratings  and  peer  ratings  on  information  discernment  behaviors. 


Self  ratings  of  performance  behaviors 


CCIR 

Filtering 

Search 

Integration 

Prediction 

Tracking 

CCIR 

-.06 

Filtering 

-.17 

Peer  ratings  of 

Search 

-.22 

performance 

Integration 

-.17 

Prediction 

-.11 

Tracking 

-.06 

Three  of  the  supplemental  items  had  interesting  correlations  with  performance  or  quality  ratings. 
One  item  was  on  the  generalizability  of  the  Azimuth  ratings  to  behavior  in  their  next  leadership  assignment. 

Self  ratings  on  generalizability  of  performance  had  a  positive  correlation  with  performance  (rcr  genz  =  .34), 
however,  their  rating  of  their  own  performance  did  not  correlate  with  performance  measured  by  observers 

(V  cr,qazs4  =  -.12). 

Self  ratings  on  another  item  (keeps  alert  for  missing,  unusual  or  unexpected  information)  had  a 
negative  correlation  with  the  integration  quality  rating  (r^ien  =  -36),  while  it  had  a  positive  correlation 
with  how  well  they  felt  they  performed  (rqazs4jaiert  =  .40).  Personnel  performing  better  did  not  report  keeping 
alert  for  missing  or  unexpected  information  as  much  as  poor  performers.  Perhaps  better  performers  felt 
that  they  already  knew  what  they  needed  to  know  while  lower  performers  tried  to  rely  on  information  that 
they  hoped  would  become  available.  This  also  relates  to  the  low  Openness  scores  for  better  performers. 
Perhaps  they  were  more  proactive  to  integrate  whatever  information  was  already  available  and  tended  to 
wait  less  for  information  to  resolve  uncertainties. 

Ratings  by  peers  on  whether  the  individual  was  "under  time  pressure  throughout  PW96”  correlated 
negatively  with  all  of  the  quality  ratings  (from  rq2Azii5  =  -.29  to  rq4>Azii5  =  -.45).  When  peers  thought  an 
individual  was  under  time  pressure,  that  individual  did  not  perform  well.  When  individuals  weren't  seen  as 
under  time  pressure,  those  individuals  performed  better.  The  appearance  of  being  pressured  is  probably  an 
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indication  that  the  individual  does  not  have  the  job  under  control.  The  peers'  rating  of  time  pressure  on  an 
individual  agreed  better  (negative  prediction)  with  performance  of  that  individual  than  ratings  of  the  quality 
of  integration. 


Combining  Measures:  Partial  Correlations 

Source  of  observations.  The  patterns  of  correlation  showed  good  agreement  for  measures  rated  by 
the  same  individuals,  but  null  to  negative  relationships  among  ratings  by  self,  peers,  and  observers.  The 
Conceptual  Capacity  Level  measure  relies  primarily  on  self-reports  and  correlated  strongly  with  the  self- 
report  version  of  Azimuth  and  the  self-report  Biodata  questionnaire,  even  on  dissimilar  items.  The  scores 
from  the  same  elements  of  the  Azimuth  self-report  and  peer  rater  versions  did  not  correlate  with  one 
another.  There  were  negative  correlations  between  self-rated  political  skills  and  seven  peer-rated  elements 
(see  Table  17).  Similarly,  there  were  hardly  any  significant  correlations  between  measures  from  observers 
and  the  self-report  measures  (from  either  Azimuth  or  Biodata)  or  the  peer  measures  from  Azimuth. 

Table  17. 

Pearson  product-moment  correlations  between  Peer  and  Self  Azimuth  elements  (sample  size  =  66, 
significant  at  .05  if  <-.24  or  >.24). 


Peer  Rated  Elements 

Self  Rated  Elements 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

12 

1  Interpersonal  focus 

.05 

-.11 

-.03 

-.17 

-.07 

-T7 

-.26 

-.06 

-.05 

-.02 

.06 

.12 

2  Team  focus 

-.09 

-.20 

-.14 

-.24 

-.10 

-.16 

-.30 

.00 

-.15 

-.07 

.03 

.01 

3  Mission  focus 

-.09 

-.22 

-.16 

-.19 

-.06 

-.18 

-.25 

-.03 

-.16 

-.12 

.00 

.09 

4  Problem  solving 

-.11 

-.18 

-.12 

-.19 

-.11 

-.14 

-.27 

-.09 

-.18 

-.16 

-.07 

-.03 

5  Knowledge 

-.07 

-.14 

-.07 

-.14 

-.10 

-.10 

-.25 

-.13 

-.12 

-.11 

-.09 

.00 

6  Planning/org  skills 

-.05 

-.14 

-.04 

-.11 

-.04 

-.07 

-.16 

-.04 

-.12 

-.12 

-.03 

.08 

7  Political  skills 

.03 

-.04 

.04 

-.05 

.08 

-.05 

-.06 

.07 

-.05 

-.01 

.10 

.08 

8  Ethics 

.05 

-.15 

-.08 

-.14 

-.14 

-.19 

-.32 

-.04 

-.09 

-.02 

.06 

.09 

9  Communic/influence 

.05 

-.15 

-.08 

-.14 

-.14 

-.19 

-.32 

-.04 

-.09 

-.02 

.06 

.09 

10  Social  maturity 

.01 

-.11 

-.06 

-.05 

-.01 

-.10 

-.20 

-.01 

-.01 

.05 

.08 

.12 

11  Self-centeredness 

.19 

.00 

.01 

-.04 

-.03 

-.05 

-.15 

.05 

.03 

.15 

.21 

.21 

12  Compulsive 

.14 

-.01 

-.03 

-.18 

-.19 

-.11 

-.15 

.02 

-.02 

.13 

.16 

.10 

On  the  six  Critical  Information  discernment  items,  there  were  no  same-item  correlations  between 
observers  and  the  self  scores  on  Azimuth,  no  same-item  correlations  between  observers  and  peer  scores  on 
Azimuth,  and  no  same-item  correlations  between  the  self  and  peer  Azimuth  scores.  It  was  anticipated  that 
there  would  be  more  agreement  among  the  different  sources  of  data  than  there  was.  This  raises  the 
question  of  whether  the  different  raters  interpreted  the  items  differently  or  whether  the  different  groups  of 
raters  had  different  patterns  of  judgment  about  the  behaviors  being  rated. 
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Multiple  Regressions 

Since  there  was  no  apparent  relationship  between  the  MCPA's  Conceptual  Capacity  Level  and  the 
performance  criterion  of  Critical  Information  discernment,  additional  variables  were  explored  for 
determining  what  affected  performance.  Several  multiple  regressions  were  computed  to  screen  variables 
and  to  retest  the  relationships  found  with  the  correlational  analysis.  (Note  that  multiple  regressions  with  a 
small  number  of  cases  are  not  generally  stable  and  are  presented  here  only  for  exploratory  purposes.) 

Variables  within  seven  different  constructs  (see  Appendix  B)  were  regressed  separately  against  the 
performance  criterion.  The  constructs  included  cognitive  complexity,  task  performance,  personality, 
leadership,  knowledge,  problem  solving  style,  and  situational  characteristics.  Stepwise  regressions  (SAS, 
1988)  were  considered  until  a  maximum  r2  was  reached,  a  maximum  number  of  specified  variables  was 
included,  or  the  ratio  of  variance  to  the  number  of  predictors  approached  one  (specifically  a  ratio  of 
Mallow's  Cp  to  the  number  of  variables). 

Cognitive  complexity.  The  use  of  cognitive  complexity  variables  by  themselves  to  predict 
performance  was  extremely  low  with  only  six  percent  of  the  variance  explained.  The  Biodata  Cognitive 
Complexity  scale  was  the  major  source  of  variance  (see  Table  18). 

Table  18. 

Multiple  regression  of  cognitive  prediction  variables  on  CID  rankings 


Cognitive  Prediction 
r*=  .06,  FM6=  2.99 

Weight 

Biodata 

Cognitive  complexity 

-3.61 

CPA  (phrases) 

Conceptual  Capacity  Level 

did  not  enter 

Azimuth  (peer  ratings) 

Azimuth  problem  solving 

Azimuth  planning  &  organizational 

did  not  enter 

did  not  enter 

Participants'  performance.  A  number  of  measures  of  performance  came  from  the  participants 
themselves  in  the  form  of  both  self  ratings  and  ratings  by  peers.  The  source  of  the  measures  came  from  the 
Azimuth  questionnaire  for  items  that  paralleled  six  of  the  Critical  Information  discernment  behaviors.  The 
proportion  of  variance  explained  by  five  self  ratings  and  three  peer  ratings  was  fairly  low  at .  17  (see  Table 
19).  Proactive  search  for  information  contributed  positively  to  the  relationship,  and  tracking  minor  events 
contributed  negatively. 
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Table  19. 

Multiple  regression  of  participants'  ratings  of  performance  on  CID  rankings 


Participants'  rating  of  performance 
r*=  .17,  F8>36=  .95 

Weight 

Azimuth  (Self  ratings) 

CCER. 

-2.98 

Filtering 

4.06 

Search 

4.65 

Integration 

-4.01 

Prediction 

did  not  enter 

Tracking  minor  events 

-4.66 

Azimuth  (Peer  ratings) 

CCIR 

did  not  enter 

Filtering 

did  not  enter 

Search 

2.47 

Integration 

did  not  enter 

Prediction 

1.39 

Tracking  minor  events 

-4.97 

Personality.  Several  personality  variables  were  used  to  examine  the  relationship  with  performance. 
Again  the  proportion  of  shared  variance  was  low  at  .15  (see  Table  20).  Traits  corresponding  positively 
were  dominance,  need  for  approval  and  peer  ratings  indicating  that  they  did  not  exhibit  compulsive 
behaviors.  Negative  contributors  were  peer  ratings  of  social  maturity  and  self  ratings  of  self-centeredness 
and  compulsive  behavior  (the  negative  direction  indicating  that  self  perception  of  high  self-centeredness  and 
high  compulsiveness  relate  to  high  performance). 
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Table  20. 

Multiple  regression  of  personality  variables  on  CID  rankings 


Personality 
^=.15,^41=  1.24 

Weight 

Biodata 

Need  for  achievement 

did  not  enter 

Dominance 

2.10 

Need  for  approval 

1.33 

Azimuth  (Self) 

Social  maturity 

did  not  enter 

Self-centeredness 

-2.93 

Compulsive  behavior 

-1.55 

Azimuth  (Peers) 

Social  maturity 

-4.65 

Self-centeredness 

did  not  enter 

Compulsive  behavior 

5.47 

Leadership.  Two  combinations  of  leadership  variables  are  presented  (see  Table  21).  The  single 
variable  of  treating  others  as  objects  explained  a  significant  proportion  of  the  variance,  but  there  was  a 
relatively  low  proportion  explained  (.15).  Extending  the  regression  to  a  total  of  four  independent  variables 
explained  a  .36  proportion  of  the  variance.  Self  ratings  of  peer  leadership  and  political  skills  related 
negatively  to  performance,  while  self  ratings  of  how  one  treats  others  and  generalizability  of  performance 
related  positively  to  performance. 
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Table  21. 

Multiple  regression  of  leadership  variables  on  CID  rankings 


Leadership 

Weight  -  1st  equation 

Weight  -  optimized  equation 

Biodata 

Peer  leadership 

did  not  enter 

-2.40 

Treats  others  as  objects 

5.73 

5.19 

Azimuth  (Self) 

Supervisory-  interpersonal 

did  not  enter 

did  not  enter 

Supervisory-mission  focus 

did  not  enter 

did  not  enter 

Political  skills 

did  not  enter 

-3.60 

Ethics 

did  not  enter 

did  not  enter 

Communication 

did  not  enter 

did  not  enter 

Supervisory-team 

did  not  enter 

did  not  enter 

Azimuth  (Supplemental) 

Performance  Applies  to  next  assignment 

did  not  enter 

5.73 

Predicted  Commander  effectiveness 

did  not  enter 

did  not  enter 

r^.15,  FM3=  7.45, p=. 0092 

i*=  .36,  F4, 44=5.72,  p=.0010 

Knowledge.  Variables  relating  to  knowledge  and  experience  were  not  very  extensive  in  this  study. 
Only  two  variables  from  the  Azimuth  element  of  tactical  and  technical  knowledge  were  regressed  against 
performance  with  a  minimal  proportion  of  variance  in  common  (.02)  (see  Table  22). 

Table  22. 

Multiple  regression  of  knowledge  variables  on  CID  rankings 


Knowledge 
r*=  .02,  F1,47=  .98 

Weight 

Azimuth  (Self) 

Tactical/technical 

-2.10 

Azimuth  (Peers) 

Tactical/technical 

did  not  enter 

Style.  The  style  of  processing  or  problem  solving  was  characterized  by  the  Biodata  Openness  scale 
and  several  individual  items  from  Azimuth.  Almost  30  percent  of  the  performance  variance  was 
contributed  negatively  by  'Openness'  and  'keeping  alert  for  unusual  information'  and  positively  by 
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adequate  analysis'  and  option  dominance  approach'  (see  Table  23).  Most  individuals  (60  percent) 
reported  a  procedural  approach,  while  20  percent  reported  a  dominance  approach. 


Table  23. 


Style 

r2=  .29,  F4,43=  4.48,  p=.0041 

Weight 

Biodata 

Openness 

-4.85 

Azimuth  (Supplemental,  Self) 

Analytic 

6.69 

Considers  own  errors 

did  not  enter 

Uncertainty-reasons 

did  not  enter 

Alert  for  unusual  info 

-5.20 

Quick,  part  solution 

did  not  enter 

Option  dominance 

3.37 

Situation.  Situational  variables  explained  about  a  quarter  of  the  shared  variance  with  performance 
(see  Table  24).  The  rank  of  position  was  the  largest  contributor  while  intelligence  jobs  and  perceptions  of 
time  pressure  contributed  negatively  to  the  relationship. 

Table  24. 

Multiple  regression  of  situational  variables  on  CID  rankings 


Situation 

i*=  .26,  F3,46=  5.28,  p=.0033 

Weight 

Questionnaire 

Command  or  staff  position 

did  not  enter 

Echelon 

did  not  enter 

Operations  job 

did  not  enter 

Intelligence  job 

-2.13 

Rank  of  position 

7.11 

Azimuth  (Supplemental,  ) 

Time  pressured  in  PW 

-3.39 

Combined  constructs.  The  variables  contributing  most  to  the  single  construct  multiple  regressions 
of  leadership,  style  and  situation  (Tables  18-24)  were  combined.  This  resulted  in  seven  predictor  variables 
with  a  shared  variance  of  .51  (see  Table  25). 
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Table  25. 

Multiple  regression  of  combined  constructs  on  CID  rankings 


Combined  constructs 
r2=  .51,  F7j39=  5.87,  p=.0001 

Weight 

Leadership 

Treats  others  as  objects 

4.03 

Performance  Applies  to  next 
assignment 

4.02 

Style 

Openness 

-4.23 

Analytic 

4.27 

Alert  for  unusual  information 

-4.52 

Situational 

Rank  of  position 

3.74 

Time  pressured  in  PW 

-1.97 

Interview  results 

Several  students  were  interviewed  to  find  out  if  there  were  any  additional  insights  that  could  be 
derived  about  the  findings.  Students  were  selected  for  interviews  if  their  scores  on  information  discernment 
and  Conceptual  Capacity  Level  were  either  high  on  both  measures,  low  on  both,  or  high  and  low.  Three  of 
the  selected  students  scored  above  average  to  high  on  both  measures  and  two  scored  low  on  both.  Two 
scored  low  on  performance  but  were  rated  among  the  highest  on  Conceptual  Capacity  Level.  Two  more 
students  were  rated  high  on  performance  and  rated  low  on  Conceptual  Capacity  Level.  One  additional 
student  was  included  because  he  ranked  best  on  performance  and  a  mid  position  on  Conceptual  Capacity 
Level.  These  students  were  asked  about  their  educational  background,  unique  assignments,  outside 
interests,  influential  people  in  their  lives,  most  notable  achievement,  and  what  job  or  position  has  been 
beyond  their  comfort  zone. 

More  similarities  were  discovered  among  these  eight  students  than  differences.  Six  of  the  eight  had 
a  father  who  had  been  in  the  military.  Six  of  the  students  listed  participating  in  sports  as  an  interest,  as 
well  as  reading. 

Differences  that  existed  did  not  clearly  delineate  differences  within  or  between  performance  and 
Conceptual  Capacity  Levels.  Those  ranking  second  through  fourth  best  in  performance  in  this  group  listed 
a  previous  Commander  as  having  a  substantial  influence  in  their  career.  The  type  of  influence  dealt 
primarily  with  observing  competence,  attitude,  problem  solving  style,  and  general  excellence.  The  five 
lower  performers  of  those  interviewed  were  commissioned  through  ROTC,  while  the  top  three  came  from 
the  US  Military  Academy  or  Officer  Candidate  School. 
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The  three  students  ranking  lowest  on  Conceptual  Capacity  Level  (scoring  7.5,  8,  and  9)  had 
bachelor  degrees  in  the  sciences  (political  science,  psychology,  and  biology),  while  the  rest-except  for  the 
highest  cognitive  ranking  individual-had  majors  in  music,  foreign  language,  personnel  management,  and 
history.  The  individual  ranking  highest  among  the  eight  on  Conceptual  Capacity  Level  was  the  only  one  of 
the  eight  who  peers  rated  very  low  on  projected  effectiveness  as  a  Battalion  commander.  The  next  three  to 
this  individual  on  Conceptual  Capacity  Level  were  the  only  ones  to  list  a  notable  achievement.  The  types 
of  achievement  were  very  different,  dealing  with  being  a  company  commander  in  Desert  Storm,  coaching 
subordinates  in  a  poor  command  climate,  and  convincing  others  to  do  things  his  way. 


Observations  of  the  Combat  Information  Center  (CIC) 

Although  the  CIC  was  not  a  specific  factor  in  the  design  of  this  case  study,  its  use  helped  to 
highlight  the  importance  of  Critical  Information  discernment.  Its  operation  was  observed  and  several 
observations  and  conclusions  are  offered. 

The  Combat  Information  Center  was  designed  to  produce  "a  single,  synthesized  picture  of  friendly 
and  enemy  forces,  and  the  battlespace  environment."  This  seemed  like  an  ideal  location  to  observe  the 
quality  of  Critical  Information  discernment.  As  it  turned  out  the  CIC  staff  did  not  do  much  Critical 
Information  discernment. 

During  the  first  few  days  of  the  exercise,  the  CIC  experienced  technical  problems  with  the 
computers.  By  the  third  day  the  system  was  mostly  working.  Even  with  the  system  operational,  the  CIC 
staff  concerned  themselves  with  fairly  mechanical  tasks  of  relaying  information,  rather  than  analyzing, 
interpreting,  and  integrating  the  meaning  of  it  (as  intended  in  the  CIC  concept,  EER,  1996).  The  CIC  staff 
were  more  concerned  with  receiving  information  and  producing  and  sending  overlays,  than  focusing  on  the 
overall  tactical  picture,  considering  possible  outcomes,  or  clarifying  the  picture  among  battlefield  operating 
systems.  They  were  interested  in  the  accuracy  of  the  information  they  received  and  at  what  echelon  to 
aggregate  it.  Their  measure  of  success  appeared  to  be  gauged  by  the  number  of  combined  overlays  they 
were  capable  of  preparing  and  the  speed  with  which  they  could  prepare  them. 

Observers'  ratings  of  their  performance  supported  these  observations.  The  CIC  staff  scored  (4.5) 
significantly  lower  than  the  rest  of  the  Mobile  Strike  Force  (6.6)  on  the  quality  of  integration  (t=  2.76, 
/K.01).  Comments  overheard  from  the  CIC  staff  also  supported  the  observation.  They  were  heard  to  say 
that  they  did  not  have  the  big  picture  of  the  battle  and  that  they  were  isolated  from  the  decision  makers. 
Therefore  they  were  not  sure  what  was  relevant.  The  MSF  Commander  was  heard  at  one  point  to  say  that 
the  CIC  product  was  not  giving  him  anything  that  he  could  use. 

The  CIC  did  not  function  as  well  as  intended.  This  could  be  attributed  to  a  number  of  factors; 
perhaps  the  most  overlooked  was  advance  considerations  to  what  functions  the  personnel  were  to  serve. 

The  design  of  the  CIC  tended  to  restrict  the  personnel  to  operators  serving  in  the  role  of  an  information 
conduit.  Their  role  as  analysts  was  not  very  strong;  and  how  they  were  to  fuse  information  from  their 
respective  battlefield  operating  systems  was  not  clearly  specified.  It  is  not  clear  whether  the  way  the  CIC 
operated  was  what  was  intended  or  the  jobs  just  evolved  this  way  due  to  the  lack  of  integration  of  people 
into  the  concept. 

The  concept  for  the  CIC  should  be  reviewed  carefully  by  those  organizations  responsible  for 
developing  it.  If  there  is  enough  merit  to  continue  to  test  it,  explicit  decisions  should  be  made  about  the  role 
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people  are  to  serve.  There  should  be  some  way  devised  to  help  the  CIC  staff  maintain  situation  awareness 
and  to  understand  or  anticipate  what  information  will  be  relevant  to  the  decision  makers. 

The  officer-in-charge  of  the  CIC  and  all  other  staff  should  be  trained  to  assume  responsibility  for 
understanding  the  battlefield  from  the  information  they  are  handling.  Additional  procedures  should  be 
implemented  for  transmitting  high  priority  alert  information  outside  the  normal  cycles  of  producing  the 
relevant  common  picture.  If  the  staff  is  not  more  responsible  for  understanding  and  using  the  information 
they  are  handling,  the  products  from  the  CIC  will  be  of  limited  value. 
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Conclusions 


The  lack  of  confirmation  of  a  relationship  between  conceptual  capacity  level  and  critical 
information  discernment  was  puzzling.  Equally  puzzling  was  the  negative  correlation  between  the  related 
Openness  scale  from  the  Biodata  instrument  and  critical  information  discernment  performance.  A 
discussion  of  possible  explanations  for  the  null  findings  follows.  A  number  of  alternatives  were  considered 
and  tested  where  data  were  available. 


Exercise  Complexity 

The  null  relationship  between  Conceptual  Capacity  Level  and  Critical  Information  discernment 
may  have  occurred  because  of  particular  conditions  of  the  exercise.  The  exercise  may  have  not  been 
challenging  enough,  therefore,  students  were  not  fully  engaged  in  completing  the  tasks.  Cognitive 
complexity  requirements  are  supposed  to  increase  at  higher  leadership  positions  in  an  organization.  Since  a 
considerable  portion  of  the  exercise  was  played  at  division  level,  one  contention  was  that  the  requirements 
for  conceptual  abilities  were  not  as  prominent  as  they  would  be  at  higher  echelons.  Problems  that  the 
students  faced  may  have  been  routine,  where  experience  dominated  and  complex  problem  solving  behaviors 
were  not  evoked.  Differences  among  the  complexities  of  different  jobs  could  have  concealed  a  direct 
relationship  between  the  predictor  and  the  criterion. 

However,  counter  to  this  argument,  there  were  several  indications  that  the  exercise  requirements 
were  challenging.  Most  of  the  students  (80%)  thought  that  difficulty  was  medium  or  higher.  Students' 
ratings  of  experienced  difficulty  moderately  correlated  with  the  criterion  but  did  not  change  the  correlation 
between  the  criterion  and  the  predictor  when  it  was  partialled  out.  Twenty-two  percent  thought  the  change 
from  expected  to  experienced  difficulty  was  positive  and  one  third  thought  it  was  undesirable.  Students 
rated  similarity  of  their  previous  experience  to  what  they  did  in  PW96.  Forty-one  students  rated  similarity 
an  average  of  4. 1,  about  the  midpoint  of  similarity.  Their  similarity  rating  did  not  correlate  with  either  the 
performance  or  Conceptual  Capacity  Level  measure,  and  partialling  out  (or  controlling  for  experience)  did 
not  increase  the  correlation  between  Conceptual  Capacity  Level  and  Critical  Information  discernment. 

Some  jobs  may  have  been  more  complex  than  others,  but  the  students'  response  to  the  job  demands 
was  probably  a  more  prominent  factor.  The  variable  of  rank  of  the  job  was  considered.  It  was  found  to 
correlate  with  the  performance  criterion  (rcr>pos=.35,  p=  .01)  but  when  partialled  out,  did  not  significantly 
improve  the  correlation  between  the  criterion  and  the  predictor. 

Most  of  the  students  had  not  performed  staff  and  command  functions  at  division  or  corps  echelons. 
Even  if  they  were  using  their  educational  experience  or  experience  from  lower  echelons,  it  seems 
inappropriate  to  say  the  problems  they  faced  were  routine.  Further  consideration  of  the  observers' 
explanations  of  Critical  Information  discernment— both  positive  and  negative-indicated  that  situations  and 
responses  were  not  routine.  All  of  these  data  show  no  indication  that  the  jobs  were  not  challenging  enough, 
too  routine,  or  too  simple. 


Measuring  Conceptual  Capacity  Level 

The  most  significant  concern  we  had  with  the  Conceptual  Capacity  Level  variable  was  the  lack  of 
difference  between  this  sample  of  CGSOC  students  who  were  mostly  majors  and  an  earlier  sample  of  Army 
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War  College  (AWC)  students  who  were  lieutenant  colonels  (Lewis,  1993).  Stratified  Systems  Theory 
suggests  that  the  distribution  of  Conceptual  Capacity  Level  should  be  different  depending  on  the  level  of 
management.  As  the  level  of  the  organization  increases,  there  should  be  a  higher  proportion  of  higher 
Conceptual  Capacity  Levels  than  at  lower  levels  of  the  organization.  There  was  no  statistical  difference 
found  between  the  two  distributions.  Several  possible  explanations  exist. 

It’s  possible  that  the  CGSOC  sample  is  disproportionately  high,  or  the  AWC  sample  was 
disproportionately  low,  although  we  found  no  specific  evidence  to  support  these  claims.  One  possible 
explanation  is  predicated  on  the  closeness  of  the  average  ages  in  each  study.  The  average  age  of  the  PW96 
subjects  was  37  and  the  ICAF  sample  was  42.  These  ages  are  close  enough  on  the  maturation  curves  in  the 
CPA  protocol  so  as  to  not  significantly  affect  current  capacity  level.  So  what  could  have  caused  this  lack 
of  difference?  Our  researchers  are  divided  on  this  question  Another  possibility  is  the  PW96  subjects 
exhibited  a  disproportionately  higher  level  of  current  conceptual  capacity  (relative  to  their  experience) 
because  the  Army  does  not  advance  people  on  the  basis  of  conceptual  capacity.  (McGee,  Jacobs,  Kilcullen, 
Barber,  1996) 

In  the  AWC  sample,  rater-rater  reliability  for  52  interviews  was  .81.  (However,  one  rater's  scoring 
was  based  on  direct  interviews,  while  the  other  rater  used  only  written  records  from  the  CPA  methods.) 

Two  AWC  instructors  rated  32  of  the  52  interviewees  on  their  potential  for  strategic  thinking  and  as  a 
general  officer.  Reliability  estimates  were  .57  and  .34,  respectively  on  these  two  measures  between  the 
instructors.  Construct  validity  was  computed  as  the  correlation  between  the  sum  of  the  instructors'  ratings 
of  potential  and  the  Conceptual  Capacity  Level  scores.  Construct  validity  was  .42  for  strategic  thinking 
potential  and  .46  for  general  officer  potential.  Although  these  validity  levels  are  somewhat  supportive,  they 
are  not  overwhelming  proof  that  Conceptual  Capacity  Level  is  a  dependable  predictor  at  present. 

Based  on  work  by  Lewis  (1993),  some  of  the  phrases  were  modified,  but  no  updates  to  the 
psychometric  qualities  have  been  made.  The  modified  phrases  measures  for  this  study  considered  objective 
scoring  items  like  length  (the  number  of  lines  of  written  explanation  of  chosen  phrases)  and  time  (the 
average  time  to  select  phrases).  Coren  and  Suedfeld  (1990)  report  that  attempts  to  use  objective  measures 
(like  the  total  number  of  words  written,  sentence  length,  and  readability  indices)  in  a  different  measure  of 
complexity,  the  Paragraph  Completion  Test,  cannot  replace  scoring  of  the  semantic  content.  Semantic 
content  was  clinically  judged  to  determine  the  Conceptual  Capacity  Level  in  this  study. 

The  MCPA  phrases  involve  self-report  preferences  for  ways  of  working  but  without  situational 
or  domain-specific  norms.  People  do  not  commonly  discuss  ways  of  thinking  and  might  not  have 
interpreted  the  phrases  as  expected.  This  is  not  something  that  would  be  apparent  in  any  of  the  validity 
checking  that  has  been  done  to  date. 

People  generally  do  not  think  about  the  way  they  process  information  and  consequently  find  it  difficult  to  respond 
meaningfully  to  questions  about  their  own  capacity  to  differentiate  and/or  to  integrate.  In  response  to  direct  questions 
about  their  information-processing  style,  individuals  tend  to  respond  in  a  socially  desirable  fashion,  simply  because  it 
sounds  good  to  be  cognitively  complex.  In  addition,  less-complex  persons  do  not  realize  that  they  actually  employ  a 
single  (or  very  few)  dimensions  when  they  process  information . . .  because  they  tend  to  assign  diverse  labels  to 
dimensional  characteristics  that  are,  in  fact,  highly  correlated,  (p.  148,  Streufert  &  Swezey,  1986). 

We  do  not  have  a  precise  language  for  how  we  think,  so  it  is  possible  there  is  wide-ranging  interpretation  of 
the  MCPA  phrases. 
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The  Symbols  portion  of  the  CPA  method  has  an  advantage  over  the  Phrases  because  it  requires  the 
individual  to  demonstrate  an  ability.  But  the  task  is  a  rule  induction  task.  This  task  would  seem  to  be  only 
partially  related  to  the  conceptual  capacity  construct.  In  the  task  the  individual  must  determine  the  rule  that 
the  experimenter  has  in  mind.  Trying  to  determine  and  conform  to  someone  else's  rule  does  not  seem  to  be 
in  total  correspondence  with  the  highest  levels  of  conceptual  capacity,  where  new  relationships  are 
recognized  and  created. 

Additional  examination  of  the  psychometric  parameters  of  the  Conceptual  Capacity  variable  seems 
to  be  in  order.  Additionally,  other  measurement  methods  should  be  identified  (or  developed)  and  tried. 
There  should  be  more  emphasis  on  requiring  the  individual  to  demonstrate  abilities  as  opposed  to  attempt  to 
describe  processes  about  which  he  or  she  may  not  be  familiar. 


Measuring  Critical  Information  Discernment 

Although  data  on  construct  validity  of  measurement  of  Critical  Information  discernment  was  not 
available,  measurement  was  focused  on  specific  behaviors.  Discernment  performance  was  examined  and 
rated  by  knowledgeable  observers.  Mostly  at  issue  was  the  breadth  of  behavior  that  the  observers  has  to 
consider  when  rating  a  Discernment  item.  To  the  extent  there  was  error  in  the  measurement,  the  observers 
probably  erred  on  the  side  of  generality,  including  behaviors  beyond  discernment.  If  anything,  a  broader 
set  of  behaviors  should  be  expected  to  be  a  more  reliable  indication  of  battle  command  performance,  not 
less  so. 


There  are  several  problems  in  measuring  cognitive  performance  by  observation,  as  we  were 
constrained  to  do  in  this  study.  These  difficulties  were  anticipated  and  means  were  taken  to  lessen  them. 
Different  Critical  Information  discernment  behaviors  were  identified.  Behaviors  that  represented  different 
interpretations  of  information  discernment  were  included,  starting  with  the  relatively  straightforward 
process  of  attending  to  and  managing  formal  information  requirements  established  by  the  commander. 
Behaviors  also  were  considered  that  were  short  of  decisions,  including  when  insightful  use  of  information 
might  occur  in  battle  command  tasks.  Critical  insights  are  characterized  by  using  information  to  make 
predictions,  keeping  alert  for  possible  dangers  that  have  been  brought  to  mind  by  some  information,  and  the 
identification  or  anticipation  of  the  need  to  make  changes  in  planned  schedules,  assets,  boundaries,  or 
missions. 

Another  possible  limitation  was  the  differences  from  observer  to  observer  in  how  a  student  was 
rated.  Differences  were  lessened  by  the  observer  training  and  by  having  members  of  the  research  team 
observe  both  the  students  and  the  observers  in  the  performance  of  their  duties.  The  narrative  explanation 
on  the  score  sheet  consistently  provided  the  most  insight  into  a  student's  Critical  Information  discernment 
performance. 

Raters  indicated  both  the  frequency  and  quality  of  the  behaviors  to  distinguish  between  the 
occurrence  of  a  behavior  and  its  quality.  The  rating  process  was  tested  in  a  prior  simulation  exercise  and 
modifications  made.  Observers  were  used  that  were  familiar  with  battle  command  tasks.  Since  the  tasks 
have  not  been  measured  in  this  way  before,  normative  or  doctrinal  standards  did  not  exist  for  comparison. 
The  score  sheet  fulfilled  its  intended  purpose.  It  enabled  researchers  to  discriminate  among  students' 
Critical  Information  discernment  behaviors,  and  did  not  impose  an  excessive  workload  on  the  observers. 
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Increased  attention  should  be  given  to  the  training  of  observers  in  future  studies.  Educating  them  of 
the  purpose  and  value  of  the  study  is  the  first  step.  Training  could  be  improved  by  having  the  observers 
rate  examples  of  the  behaviors,  have  them  distinguish  among  behaviors,  and  then  discuss  ratings  from 
different  observers.  This  process  was  approximated  with  the  trial  application  in  the  MSF  simulation 
exercise,  but  only  about  a  fourth  of  the  observers  were  available  to  participate.  Twenty-two  of  the  thirty 
observers  went  through  a  one  hour  training  orientation  prior  to  the  exercise.  The  remaining  eight  received 
individual  training  once  the  exercise  began.  More  deliberate  training  would  undoubtedly  help  give 
observers  more  precise  ideas  about  what  to  look  for  and  what  to  record. 


Individual  and  Group  Behaviors 

The  Critical  Information  discernment  behaviors  that  were  observed  were  assumed  to  be  attributable 
to  the  individual  being  rated.  However,  it  was  recognized  that  an  observer  might  only  hear  what  the 
individual  is  repeating  from  what  others  have  told  them.  An  individual  in  a  command  post  is  not  working 
in  isolation,  but  is  processing  information  and  having  ideas  triggered  from  what  others  say.  The  dynamic 
environment  introduced  other  factors  that  were  not  assessed  in  this  case  study.  Factors  such  as  group 
cohesion  and  the  lack  of  an  actual  rank  hierarchy  limited  the  interpretation  of  these  findings  as 
generalizable  to  tactical/troop  units.  Future  experiments  could  examine  individual  ability  to  discern  critical 
information,  perhaps  by  assessing  a  controlled  task  in  conjunction  with  a  tactics  class  or  inserting  and 
tracking  probes  in  Combat  Training  Center  rotations  and  Division  and  Corps  Warfighter  exercises. 


Additional  Measures 

There  was  little  information  obtained  indicating  knowledge,  experience,  or  motivation.  These 
should  be  strong  individual  difference  variables  that  can  help  account  for  many  of  the  differences  in 
performance. 

Experience.  A  plausible  hypothesis  is  that  experience  or  knowledge  had  a  considerable  influence 
on  performance.  Differences  in  experience  could  have  overshadowed  other  differences  or  effects  of 
cognitive  complexity  or  others.  Even  with  Azimuth  ratings  on  tactical  and  technical  knowledge  and  the 
additional  survey  question  on  similarity,  there  was  relatively  little  information  obtained  on  what  each 
student  knew  and  what  kind  of  tactical  and  operational  experience  he  or  she  had. 

Motivation.  If  we  assume  motivation  is  higher  when  the  level  of  challenge  is  roughly  equal  to  the 
level  of  ability  (conceptual  capacity),  then  lower  motivation  would  be  in  evidence  for  high  capability  and 
low  capability  people  if  the  tasks  are  construed  as  moderately  difficult.  The  high  capability  people 
disengage  out  of  boredom,  and  the  low  capability  people  disengage  because  they  can’t  keep  up.  We 
attempted  to  account  for  these  two  categories  in  several  ways.  At  first  we  partialled  out  estimates  of 
motivation  on  25  students.  There  was  no  significant  improvement  in  the  magnitude  of  the  correlation 
between  performance  and  conceptual  capacity  level  (rciD,cLmot  =  .07).  We  also  examined  the  partial 
correlation  between  performance  and  Conceptual  Capacity  Level  when  difficulty  was  controlled  for 
(r  ciD,cL.diff=  -.04,  p  =.85).  Motivation  and  experience  should  be  considered  to  a  greater  extent  in  future 
studies  in  this  area. 
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Recommendations 


•  Retest  the  hypothesis  with  other  objective  measures  of  conceptual  capacity.  The  existing  CPA  or 
MCPA  could  be  used  again  so  it  can  be  compared  against  other  methods.  Consider  augmenting  the 
CPA/MCPA  with  additional  tools  that  measure  other  aspects  of  conceptual  capacity.  In  addition  to 
self-report  or  concept  formation  measures,  measures  should  be  identified  or  developed  that  address  the 
abilities  to  understand  complex  and  dynamic  relationships  and  frame  solutions  for  these  problems. 

•  Additional  study  is  needed  to  understand  the  relationship  between  conceptual  capacity  and  individual 
performance.  The  hypothesis  should  be  retested  with  a  common  task  that  all  individuals  do  by 
themselves.  The  task  should  be  one  where  good  measures  on  performance  of  information  discernment 
and  related  battlefield  visualization  skills  can  be  taken.  Additionally,  we  should  consider  measuring 
performance  in  mature  organizations.  If  proper  controls  can  be  developed,  an  experiment  conducted  in 
a  tactical/troop  unit  would  probably  yield  the  most  useful  information. 

•  The  measurement  of  Critical  Information  discernment  should  be  developed  further,  especially  in  the 
definition  of  desirable  and  undesirable  behaviors  and  descriptive  anchors  used  in  measurement. 
Develop  training  techniques  which  provide  observers  the  required  expertise  for  accurately 
distinguishing  among  behaviors.  Additional  ways  of  measuring  the  human  dimension  of  battlefield 
visualization  and  decision  making  should  also  be  considered. 

•  Future  studies  should  expand  the  hypothesis  to  consider  additional  skills  related  to  a  leader's  conceptual 
competence.  Critical  Information  discernment  is  only  one  element  of  battlefield  visualization.  It  tends 
to  be  more  of  a  convergent,  closed-ended  task  rather  than  a  divergent  one,  where  creative  problem 
finding  and  novel  solutions  are  important. 

•  A  systems  approach  could  be  followed  to  better  consider  the  whole  person  and  environmental 
influences  on  them.  Motivation  and  experience  are  two  areas  recommended  for  better  measurement. 


Summary 

The  findings  in  this  study  underline  the  complexity  involved  in  understanding  the  cognitive  aspects 
of  human  behavior  and  the  intricate  relationships  among  the  factors.  The  findings  emphasize  the  need  for 
increased  effort  to  understand  and  define  the  conceptual  skills  that  are  desired  in  Army  leaders.  Measuring 
conceptual  capacity  in  a  way  that  requires  individuals  to  demonstrate  their  ability  provides  a  possible  focus 
for  future  research  and  training  development  efforts.  Follow-up  studies  are  essential  to  address  these 
additional  aspects  of  the  human  dimension  of  Battlefield  Visualization  and  to  assist  in  the  design  of 
strategies  which  address  the  development  of  conceptual  competencies.  The  best  possible  outcome  from  this 
study  would  be  that  it  energizes  multiple  efforts  for  improved  leader  development  programs. 
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Appendix  A 

Critical  Information  Discernment 
Score  Sheet 


A-l 


1 .  Observer _  Critical  Information  Observation  Sheet 

2.  Student _  3 .  Date 


activity  when  he  or  she  should  have.  &  Quality?-.  1 0  -  One  of  the  best 


Appendix  B 
Number  of  Participants 
Completing  Each  Measurement  Instrument 


B-l 


Number  of  Participants  Completing  Each  Measurement  Instrument  from  each  CGSOC  student  group 


Measurement 

MSF 

Student  Corps 

Total 

Modified  CPA 

39 

26 

65 

Critical  Information  Discernment 

33 

17 

Biodata  Survey 

40 

27 

67 

Azimuth  Survey 

41 

27 

68 

Experience  follow-up  survey 

28 

17 

45 

Estimate  of  motivation 

19 

6 

25 

Interview 

8 

0 

8 

All  study  participants 

41 

27 

68 

Appendix  C 


Categories  of  Measures  Used  in  Prairie  Warrior  96 


Cognitive  prediction  mean_CL  (average  complexity  level)  CPA  methodology,  modified  for  this  study.  Self-report  and  CPS  scorer 

interpretation. 


C-3 


qazl-qaz6,  qazsl-qazs6  Azimuth  supplemental  items.  Self  &  peer  reports. 


C-4 


